Skip to content

Extend datatypes in python API and TUI#461

Merged
JoeZiminski merged 26 commits intomainfrom
extend_datatypes
Feb 21, 2025
Merged

Extend datatypes in python API and TUI#461
JoeZiminski merged 26 commits intomainfrom
extend_datatypes

Conversation

@JoeZiminski
Copy link
Member

@JoeZiminski JoeZiminski commented Feb 10, 2025

Expose all 'Narrow' datatypes from Neuroblueprint. This basically extends the allowed datatype names (e.g. ephys is broad, icephys is narrow).

This PR adds:

  1. documents a new set of narrow datatypes in canonical_configs which match the neuroblueprint narrow datatypes
  2. allows creation and transfer of narrow-datatype foldernames.
  3. Adds a new menu to the TUI to select the datatype checkboxes to display. These configs are saved per-project, and conversion to the new settings under-the-hood occurs so the changes update silently across versions.
  4. All tests used to use the broad datatype. The tests are refactored so they still all only use the broad datatype. New tests specifically for narrow datatypes are added.
  5. Documentation is not added yet. The screen (should be) quite intuitive, and a lot of pictures of the TUI need to be retaken to include this new button. This can be done in the large upcoming docs improvement [Documentation] Improve documentation #207 (comment added to issue) where it will be clearer how this slots in.

In NeuroBlueprint, it is not allowed to mix broad and narrow datatypes. This is not validated in this PR, but will be added to #464.

How to test

TUI

In the TUI, on the 'Create Folders' tab and 'Transfer' (Custom) tab there is a new button 'Displayed datatypes'. Clicking this will open a new window where you can select which datatype checkboxes to display. It will be good to know a) this works as expected b) is intuitive to use. Which checkboxes are displayed and their status should be saved across datashuttle sessions.

Python API

You can try creating folders like:

from datashuttle import DataShuttle

project = DataShuttle("my_first_project")

project.make_config_file(local_path="....")  # if not done already

project.create_folders("rawdata", "sub-001", "ses-001", "cscope")  # something wrong like "cscopex" would fail

You could also try adding a file to the "cscope" folder and doing something like:

project.update_config_file(central_path="path to central project on local / mounted filesystem", connection_method="ssh")

project.upload_custom("rawdata", "all", "all", "cscope")

@JoeZiminski JoeZiminski force-pushed the extend_datatypes branch 4 times, most recently from e9b82e0 to 4550a4f Compare February 12, 2025 09:57
@viktorpm
Copy link

viktorpm commented Feb 19, 2025

I tested both the TUI and the Python API (from a Python console started in the terminal). I have a couple of suggestions/observations:

  • It would be useful to see what the individual data types mean when I hover over them.
  • I'm not entirely sure what "transfer" is supposed to do. I tried it both locally and via SSH. I managed to establish a connection and attempted to transfer files here: /nfs/nhome/live/vplattner/datashuttle_transfer_test/test_project.
    UPDATE: I initially tested this function with empty folders, which wouldn't normally happen. It works as expected when there are files in the directories.

@JoeZiminski JoeZiminski requested a review from viktorpm February 20, 2025 10:59
@JoeZiminski JoeZiminski merged commit 3ae3d5d into main Feb 21, 2025
16 checks passed
@JoeZiminski JoeZiminski deleted the extend_datatypes branch February 21, 2025 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants