loss NaN during the training

[PrototypeNx]

Hi, I tried to apply the PRoPE to the NVS task.
In my implementation, I change the RoPE to PRoPE in Qwen-Image 20B model and finetune it from the pretrained model using flow matching loss.
However, NaN always occurs after thousands of steps of training, and I checked the dataset and the camera params.
I wonder if the PRoPE is suitable under flow matching training.

[peelstnac]

I experience a similar issue. I think the discussion about camera extrinsic scale may be relevant. In my case, it may be that sometimes my scales are too small which may cause precision issue. I will update if I manage to fix this.

Update: FYI, this fixed the issue.