Super slow fit() function on HPC cluster?

[jordandeakin]

Hi all 🙂

I’m aiming to fit a model to multiple participants in parallel on our university’s HPC cluster. On my local PC, fitting one dataset takes about 5 minutes, but on the cluster, the progress bar estimates ~4 days for the same task/data.

I’ve confirmed that both my Julia and Unfold versions are identical to my local setup.
I’ve also tried:

• Setting BLAS.set_num_threads(1) (no improvement)

• Moving my Julia depot to an SSD-based file system (as recommended by the cluster team)

I can also run some tutorial data (specifically this. It runs a bit slower (perhaps a minute or so) than on my local PC, but definitely not 4 days, haha!

Has anyone encountered something similar or have suggestions for what could cause such a slowdown? I can provide my code if needed.

Thanks in advance!

[behinger]

Uff - that should definitely not be the case. Which unfold version are you using?

Edit: It is also strange that the tutorial takes so long in the first place.

Can you run typeof(data) for your own data?

[jordandeakin]

thanks for your fast reply! The Unfold version is v0.8.7.

typeof(data) returns Matrix{Float64} ( I want to fit to multiple channels to later do TFCE)

I read it in from an EEGLAB .set file with PyMNE and then turn it into a Julia array with

 eeglab = PyMNE.io.read_raw_eeglab(loadName)
    data = eeglab.get_data()

    print("Data Loaded\n")

    # Turn into a Julia array
    data = Array(PyArray(data))

I've also tried to see what is going on using Profiler (when running the tutorial data) but the output is huge and a bit hard to interpret in the terminal.

[behinger]

Looks good.

the tutorial one minute is already very slow for you... (The fit alone should take less than 100msec (at least the second time you run it - will doublecheck on Monday ). This confuses me the most atm.

Can you check whether a simpler formula, e.g. just intercept is fast?

Also those things will be helpful:

• how many channels
• what's the formula
• what is size(data)
• are continuous columns (if any) casted as float/integer?
• you do get data without specifying units. I think this should be fine, but please check whether the scale of your data is not off by 10e6 (mne returns in V not uV)

Edit: regarding profile, i'd recommend @profview, this will give you a flame graph - maybe easier to understand (screenshot could be helpful too... )

[jordandeakin]

how many channels

• There are 63 channels,

what's the formula

• The formula is
  design = [
  "fixation" => (@formula(0 ~ 1 + VA + VO + spl(sac_ampltide,5)), firbasis(τ=(-0.2, 0.8), sfreq=500))
  ]
  (but changing it to just @formula(0 ~ 1) is also slow (I can't say how slow yet because it hasn't printed the progress bar, it takes a long time to get to that part).

what is size(data)

• size(data) = (63, 2434390) (it was a long experiment)

__are continuous columns (if any) casted as float/integer?_

• yes, after the events are read in, I make sure they are Float64

    EEG = MAT.matread(loadName)
    evts = DataFrame([k => dropdims(v, dims=1) for (k, v) in EEG["event"]])

    evts.sac_amplitude = Float64.(evts.sac_amplitude) 
    evts.VA = Float64.(evts.VA)
    evts.VO = Float64.(evts.VO) 

I've also tried getting rid of all other columns/events (the events data frame looks like this after) Edit: I realised latency isn't Float64, but I just changed that now

you do get data without specifying units. I think this should be fine, but please check whether the scale of your data is not off by 10e6 (mne returns in V not uV)

• The scale I get is V, but I turn it back into uV after I change it into a julia array

   # Turn into a Julia array
    data = Array(PyArray(data))
    data = data * 1e6 #Note, mne uses SI units so we need to multiply the data by 1e6 to get back into microvolts.
    data = data[selectedChans, :] # Selected Channel -

[jordandeakin]

Hi,
I just fixed it. It was latency not being Int64. I thought I had changed it but I didn't save the change (sorry!). Something about my local setup seems to ignore this.

Thanks so much :D

[behinger]

Oh wow. Thanks for reporting that, I will enforce that in a future version of unfold!

Super glad it is now running fast 💪🐆