Questions About Training from Scratch and Reproducing Paper Results

[wujunnan0929]

Hi, thank you for this excellent work! We're very interested in your approach and have been working to reproduce your results. We encountered a few issues we'd appreciate your guidance on:

1. Training from scratch:

It seems like lack of Phase 1 training code
Could you provide instructions for training the model without using pretrained weights?
Are there any specific hyperparameter settings or training procedures we should follow?

2. Reproduction with pretrained weights:

We're observing slightly lower performance compared to the paper's reported results on Mini-ImageNet:

Paper results (Session-wise Harmonic-mean): 68.71, 63.87, 60.94, 57.98, 55.27, 52.41, 52.68, 53.12, 58.12
Our results: 65.774, 62.872, 59.722, 56.443, 54.869, 51.634, 52.202, 52.207

Could you suggest any potential reasons for this performance gap?
Are there any additional training strategies or implementation details that might help achieve results closer to those reported in the paper?

We'd be very grateful for any insights you could share. Thank you for your time and for making this work available to the community!

[noorahmedds]

Hi @wujunnan0929,

Thanks for raising this concern. The instructions were not previously very clear for base training. I have added more instructions for the pretraining phase and some new code that should ameliorate this. Please follow the instructions in the Readme (under "Pretrain Session") and let me know how it fares for you.

[wujunnan0929]

Thanks for the update!

About point 1: We got the base session model trained from scratch! It would be awesome if you could also share the training config for CUB-200.

About point 2: We tried the pretrained weights, but are still seeing the same gap with the paper's results. Any other ideas on what we might be missing? We'd appreciate any hints!

Thanks again for your help!