TRX format

[neurolabusc]

@skoudoro providing validation TRX datasets are a terrific idea. I think this is a terrific contribution. Please take the tone of my comments as enthusiastic. However, I do think that the current expected.trx is unusual, and not what I expected based on the TRX specification. Perhaps @frheault can comment. Here are the contents of your TRX file:

dpg
dpv
header.json
positions.3.float16
offsets.uint64
./dpg/g_AF_L:
  dpg_AF_L_mean_fa.float32
  dpg_AF_L_volume.float32
./dpg/g_AF_R:
  dpg_AF_R_mean_fa.float32
./dps:
  dps_algo.uint8 dps_cw.float64
./dpv:
  dpv_cx.uint8 dpv_cy.uint8
  dpv_cz.uint8
./groups:
  g_AF_L.int32
  g_AF_R.int32
  g_CST_L.int32

Using the prefixes dps_, dpv_ and dpg_ is not described in the specification. For the per-stream and per-vertex types this simply just refers to the naming, so its not a big issue. dps and dpv are exhaustive, defining the values for each streamline and vertex. Therefore, a visualization tools can just describe the feature as dpv_cx instead of cx. However, the dpg coding can be sparse: you do not have to specify all groups, and the same streamlines might be associated with different groups. So while I think your implementation is legal, I do not think it is the obvious definition provided in the specification. I would expect "/dpg/g_AF_L/mean_fa.float32" and "/dpg/g_AF_R/mean_fa.float32" instead of "/dpg/g_AF_L/dpg_AF_L_mean_fa.float32" and "/dpg/g_AF_R/dpg_AF_R_mean_fa.float32". In other words, the property "mean_fa" is shared for different groups, rather than different properties for each group. The method you use will probably confuse subsequent processing tools and result in visualization tools showing an overwhelming number of properties.

As an aside, for a reference implementation I would use offsets.uint32 instead of offsets.uint64 for tractgrams with fewer than 2-billion vertices. The positions.3.float16 is also valid, but degrades performance and is not a native datatype for many languages (requiring a translation). I also note that the data is saved as an uncompressed zip file. Again, perhaps the idea for this sample data is to explicitly demonstrate edge case to validate tools rather than best practices, though my sense is that you might want to distribute a compressed file and outline command line methods to generate valid variations of the mesh that illustrate different storage/compression. For example, with the provided data:

zip -9 expectedz.trx expected.trx
tff manipulate-dtype expected.trx expected_p32o32.trx --positions-dtype float32 --offsets-dtype uint32
zip -9 expected_p32o32z.trx expected_p32o32.trx
tff manipulate-dtype expected.trx expected_p16o32.trx --positions-dtype float16 --offsets-dtype uint32
zip -9 expected_p16o32z.trx expected_p16o32.trx

[skoudoro]

Very great and exciting feeback! thanks @neurolabusc!

These files has been initially created by @frheault for unit testing the python package (so testing edge cases)

My thought here was:

1. No reason for these files to be used only in the python packages, they should be obviously visible and available for every languages.
2. Start to create more standard test files (exactly like you propose) and document it
3. Make sure that all languages that implement TRX use those files in their tests. I would like to avoid disparency in TRX eco-systems.

I will try to do a push on this for 2026 when I find time. I believe TRX should be used much more but I also know that every new adoption take times...

[frheault]

Hello to both, like @skoudoro mentionned this file is much more oriented for unit test and the filenames have extra information compared to the specification only to easily verify what is loaded and overall sanity checks. It also does not aim to optimize read time, but to use many datatypes and functionnalities.

In a real file generated for a real analysis I would have use the following:

dpg
dpv
header.json
positions.3.float32
offsets.uint32
./dpg/AF_L:
  mean_fa.float32
  volume.float32
./dpg/AF_R:
  mean_fa.float32
  volume.float32 # I believe in real life having consistent dpg across groups is important, even if optional
./dps:
  algo.uint8
  cw.float64
./dpv:
  colors.3.uint8 # These 3 colors could also represented as a 3D array instead
./groups:
  AF_L.int32
  AF_R.int32
  CST_L.int32

But the expected.trx was not made to be that close to reality, I simply want a file that is valid, but easy to double check.