MEG - Add support for parallel execution in slurm

[allermat]

          Hi @RikHenson, @olafhauk, @dcdace 

I thought about this, specifically the parallel execution. Below is an example data structure from this mne-bids tutorial:

|MNE-sample-data-bids\
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- sub-01\
|------ ses-01\
|--------- sub-01_ses-01_scans.tsv
|--------- meg\
|------------ sub-01_ses-01_acq-calibration_meg.dat
|------------ sub-01_ses-01_acq-crosstalk_meg.fif
|------------ sub-01_ses-01_coordsystem.json
|------------ sub-01_ses-01_task-audiovisual_run-1_channels.tsv
|------------ sub-01_ses-01_task-audiovisual_run-1_events.json
|------------ sub-01_ses-01_task-audiovisual_run-1_events.tsv
|------------ sub-01_ses-01_task-audiovisual_run-1_meg.fif
|------------ sub-01_ses-01_task-audiovisual_run-1_meg.json
|--- sub-emptyroom\
|------ ses-20021206\
|--------- sub-emptyroom_ses-20021206_scans.tsv
|--------- meg\
|------------ sub-emptyroom_ses-20021206_task-noise_channels.tsv
|------------ sub-emptyroom_ses-20021206_task-noise_meg.fif
|------------ sub-emptyroom_ses-20021206_task-noise_meg.json

When we call mne_bids.write_raw_bids it writes the current fif file into the BIDS folder, but also updates several other files, (e.g. the participants.json, participants.tsv, and the *_scans.tsv file). If we execute the code in parallel, then there is a chance that two processes attempt to write the same file at the same time, which would lead to an error. If we parallelize across subjects then these would be the participants.json and .tsv files.

I'm sure there's a way of implementing this so that it works in parallel, the mne-bids-pipeline has even more complicated dependencies between files and it works fine in parallel. But looking at their implementation of parallelization I feel I'm out of my depht there.

Do you have any thoughts on this?

Thanks,
Máté

Originally posted by @allermat in #4 (comment)