RDMs.save() fails if noise estimates have different sizes #454
Description
Thanks for considering reporting an issue in rsatoolbox to us! Below are some tips that make it easier for us to understand what happened.
Describe the bug
If an RDMs object has noise estimates that don't have identical sizes, perhaps multiple subject RDMs where subjects' ROIs vary, saving to hdf5 fails when it tries to turn the list of those estimates into a single numpy array.
To Reproduce
import rsatoolbox as rsa
testRDM = rsa.rdm.rdms.RDMs(np.random.rand(2,10,10),'crossnobis',rdm_descriptors={'noise':[np.random.rand(5,5), np.random.rand(5,5)]})
testRDM.save('test.hdf5') # Succeeds
testRDM = rsa.rdm.rdms.RDMs(np.random.rand(2,10,10),'crossnobis',rdm_descriptors={'noise':[np.random.rand(5,5), np.random.rand(6,6)]})
testRDM.save(test2.hdf5') # Fails
Expected behavior
Successful saving.
Versions
Please report:
rsatoolbox 0.3.0
python 3.11.11
Additional context
In my actual use case, I ran into this trying to save the output from calc_rdm, which in turn was using a list of Datasets and list of noise estimates. The noise estimates were generated from rsa.data.noise.prec_from_unbalanced() using the same Datasets.
stack trace
ValueError Traceback (most recent call last)
Cell In[88], line 1
----> 1 testRDM.save('test2.hdf5')
File .venv/lib64/python3.11/site-packages/rsatoolbox/rdm/rdms.py:386, in RDMs.save(self, filename, file_type, overwrite)
384 remove_file(filename)
385 if file_type == 'hdf5':
--> 386 write_dict_hdf5(filename, rdm_dict)
387 elif file_type == 'pkl':
388 write_dict_pkl(filename, rdm_dict)
File .venv/lib64/python3.11/site-packages/rsatoolbox/io/hdf5.py:30, in write_dict_hdf5(fhandle, dictionary)
28 file = File(fhandle, 'a')
29 file.attrs['rsatoolbox_version'] = version('rsatoolbox')
---> 30 _write_to_group(file, dictionary)
File .venv/lib64/python3.11/site-packages/rsatoolbox/io/hdf5.py:50, in _write_to_group(group, dictionary)
48 elif isinstance(value, dict):
49 subgroup = group.create_group(key)
---> 50 _write_to_group(subgroup, value)
51 elif value is None:
52 group[key] = Empty("f")
File .venv/lib64/python3.11/site-packages/rsatoolbox/io/hdf5.py:47, in _write_to_group(group, dictionary)
45 group[key] = value
46 elif isinstance(value, list):
---> 47 _write_list(group, key, value)
48 elif isinstance(value, dict):
49 subgroup = group.create_group(key)
File .venv/lib64/python3.11/site-packages/rsatoolbox/io/hdf5.py:74, in _write_list(group, key, value)
61 """
62 writes a list to a hdf5 file. First tries conversion to np.array.
63 If this fails the list is converted to a dict with integer keys.
(...) 71 list to be written
72 """
73 try:
---> 74 value = np.array(value)
75 if str(value.dtype)[:2] == '<U':
76 group[key] = value.astype('S')
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.```