Clarification on Two Training Modes in train_stage1.sh

[zxcbb16]

Description

I'm working with the train_stage1.sh script for the first stage of TabICL curriculum learning and noticed it provides two training approaches:

1. Training with on-the-fly generation of prior datasets
2. Training with pre-saved prior datasets (generate first, then load and train)

My Question

Could you clarify whether these two approaches are:

• Required to run together (i.e., execute both for complete Stage 1 training), or
• Alternative options (i.e., choose only one to complete Stage 1 training)?

Thank you for your help!

[dholzmueller]

Hi, these are alternative options. We used the pre-saved prior datasets approach for pre-training, as the on-the-fly generation could be slower than the GPU steps, so you wouldn't fully utilize the GPUs on GPU nodes. In the pre-saved approach, you can run the pre-generation on a CPU node.

[zxcbb16]

Thanks for clarifying the training approaches!
I just noticed a small point to confirm: the paper mentions 160k steps for Stage 1 training, but the code seems to set it as 100k steps. Could you explain this difference?
Thanks!

[dholzmueller]

I suspect this is because we didn't train the model all at once and only realized later that we actually used 160k steps in total and not 100k steps.