Man, I was almost ready to give up on this one. I had been trying to find the cause of a very peculiar bug for months now without getting anywhere. It occasionally caused all results to be dropped without any error, deadlock or exception to trace down. Because it only happened occasionally, under varying conditions, I strongly suspected something at the threading level was going wrong. So I have been staring at the same 30 (or so) lines of C# code for months now, trying to figure out where it could drop something or do anything wrong. (strong feelings of resentment swell as I think about those lines). Of course, the latest speed improvements made it impossible to continue without figuring out what was going on.
Don’t ask me how I figured it out, but yesterday, I decided nothing could be wrong with those lines of code and instead, something in the neural code has to be wrong, more specifically, with the variables. So I built myself a better view on the memory content of the processors, and lo and behold, there they are: 3 innocent globals that get carried over from one algorithm to the next, causing all sorts of mayhem. In total, the fix took about 20 seconds. Which leaves me with mixed feelings: I’m soooo glad I finally found it. If only I had looked in the right place 3 months ago.