Error when loading large (>2GB) datasets from slow media (harddrive)

[TomVCoopmans]

Hello,

I am running into issues with large TIFFs (~16 GB), even though the dataset does not exceed free RAM (~20 GB).
Loading in the raw images from the .LIF files (From Leica SP5) as well as converting both the raw .LIF as well as the extracted TIFF into XML (~6.5 GB) using BigDataViewer causes the plugin to crash when loading the data.

Remarkably, the images (both LIF and XML) open smoothly with BigDataViewer.

Errors mention "Exception in thread "AWT-EventQueue-0" com.jogamp.opengl.GLException: Caught IllegalArgumentException: null on thread AWT-EventQueue-0".

I first had the errors on 0.5.2, and I since updated to 0.7.0, with a fresh install of FIJI, fully updated and no other plugins except BioFormats the errors remain. All settings are default settings.
BigTrace (both 0.7.0 and 0.5.2) work fine with smaller files (tested with TIFFs up to 2 GB, tiled stack of 88 Z slices).

Changing GPU (Dedicated NVIDIA or Intel) does not matter. Errors also appear on a different PC (Both Win11).

What would be the next steps to take?

Thank you,

Tom

[ekatrukha]

Hello Tom,

Thanks for the time to log an issue.
Strange that it does not work with XML.

How much GPU memory do you have?

Is it 32-bit data maybe?

I would first try to increase allowed GPU memory here to 90% of GPU and restart FIJI.

For NVIDIA I would even try to set it larger than only GPU memory, I think they redirect RAM.

Cheers,
Eugene

[TomVCoopmans]

Hi Eugene,

I checked, and the data is 8-bit. I have 24 out of 32 GB RAM available when Fiji idles, with 12 GB dedicated and 16GB shared RAM for the GPU (so 28 GB GPU RAM according to the performance tab). I changed my laptops power mode, forced the NVIDIA GPU and plugged it in to remove any energy saving features that might limit performance.

I set the allowed GPU memory to 16000, with the cache size to 64mb and this allows me to open a 6 GB image in a 28 GB LIF file, but performance is very low. Opening the 6GB TIFF extracted from the LIF does still crash.

Opening the XML does not work, I will look into this tomorrow.

Thanks for the fast response,

Tom

[TomVCoopmans]

Hi Eugene,

I think I found the error. The files I tried to open were located on an external hard drive. When I copied them to my internal SSD, the errors disappeared and I could open the large LIF files and XMLs. I think the slow read speeds caused an error.

The strange thing I is still that opening the LIFs and XMLs from the external hard disk does work with BigDataViewer. So the error may be due to the BDV playground used by BigTrace.

Thanks for the help,

Tom

[ekatrukha]

Hello Tom,

ah, I see, interesting.

Indeed, BigTrace runs on top of BVV playground (Big Volume Viewer fork)
and since it shows full 3D view,
it requires more data than Big Data Viewer (single slice view).

But it should handle the case of slow reading speed correctly.

I think I also noticed this type of behaviour before,
so now I will try to investigate in details.
For now, I will close this issue and open a new issue in BVV-playground.

Cheers,
Eugene