3.3.4-dev - DJV seems not to free substantial amount of memory when all files are closed

[aberkl]

DO:

• start DJV and check memory usage
• load file(s) and check memory usage
• close all files and check memory usage

HAPPEN:

DJV started, no files loaded    149.793 MB              
File(s) loaded                  2005.04 MB              
All files closed                1681.17 MB

SHOULD:

• DJV should free all previously allocated memory

Compare to #682.

[darbyjohnston]

Thanks for reporting this; I found a memory leak, but I think most of this is actually due to secondary caches growing too large. There are a number of internal caches for things like thumbnails and rendering that were probably set too high. I lowered the values which will help and hopefully won't affect performance too much. I'll update the issue when I have the memory leak fixed.

[darbyjohnston]

OK, I think the memory leak is fixed and I also added a couple of features to help with troubleshooting.

There is a new "diagnostics" tool that displays how many objects are currently allocated:

There is also a new command line flag that starts a loop and continuously opens and closes any files given on the command line. I let this run for about 20 minutes watching the object allocations and it remained stable after the fixes.

One thing to note about the diagnostics tool is the number of images includes things like icons and character glyphs, so even when no files are loaded it will show a couple of hundred images. These are generally very small though, like 20x20 pixels for the icons.

The widget count shown in the tool is also quite high, instead of creating all the widgets at startup I should probably try and create them only when needed. That would help performance and also memory usage a bit.

[aberkl]

Not sure this will solve low-memory problems and freezing a system: Set cache in percent of total RAM. But it could probably make it less likely:

[darbyjohnston]

Hmm, this is interesting:
https://lemire.me/blog/2020/03/03/calling-free-or-delete/

It looks like even though a program may free memory, it might not immediately get released, or may not show as being released. But maybe that's Linux specific, since the memory does go up and down in the Windows system monitor.

[aberkl]

After compiling 3.3.4-dev from yesterday's sources, I have the feeling memory usage is far less critical. So far I have not encountered any system freezes caused by "out-of-memory".

The only thing I still don´t really get is the fact when closing all loaded files, DJV still holds massively more RAM compared to "open with no files yet loaded".

But #683 (comment) maybe explains this.

Process 'djv' memory usage: 6945.96 MB          # a bunch of files loaded
Process 'djv' memory usage: 4005.11 MB          # all files closed
Process 'djv' not found.                        # DJV closed
Process 'djv' memory usage: 170.27 MB           # DJV started with no files

Will give it further tests.