The video memory usage of pygenn

[OOOOOmoi]

I was exploring the maximum model size that pygenn can run with 80G of video memory space, and for that I counted the video memory usage under different scenarios (all matrix types are PROCEDURAL), as shown in the table below:

Column A represents the num_recording_timesteps parameter I set during model.load. When the number of neurons goes from 1.3e07 to 2.6e07, the video memory usage increases linearly from 5G to 8G, like the 2nd row compared to the 3rd row. Strangely, when I transition from 2.6e07 to 3.2e07, the video memory usage doesn't seem to increase linearly, but exponentially from 8G to 47G.
So I am rather questioning why this exponential growth is occurring, I tried to scale down the synapses at 3.2e07 scale, the matrix type is PROCEDURAL, so I don't think that scaling down the synapses will reduce the memory footprint, but no matter how much I scale the synapses down, the video memory usage doesn't go down as shown in rows 6~8.
So I guess it has something to do with the number of threads supported by CUDA itself. However, when I linearly scale up the number of neurons by a factor of 10 for a model with a neuron count of 2. 6e07, the video memory usage of the model at 2. 6e08 is about 70G, which seems to show a linear increase as shown in rows 3&4. However, 2. 6e08, which is much larger than 3. 2e7, only consumes about twice the video memory usage of the latter. What do you think about this?

[tnowotny]

To be honest, it looks a bit mysterious indeed. Looking at the examples from your table, it must have to do with the number of neuron and synapse populations more than the number of neurons per se but I honestly don't know why that would be the case.
That the number of synapses doesn't matter is expected for procedural and that seems to be confirmed in your experiments.

[neworderofjamie]

Are these models definitely being simulated correctly at all scales?

[OOOOOmoi]

Yes, as long as the video memory limit is not exceeded, it can be emulated successfully.

[neworderofjamie]

Good to know! If you want to know where all the memory is going you can try increasing the log-level by passing runtime_log_level=PlogSeverity.DEBUG to the GeNNModel constructor (after importing PlogSeverity with something like from pygenn import PlogSeverity). You will get a lot of output but it should include lines for each array allocated.