Isn't min-SNR training strategy counter-intuitive?

[AbrightWay]

Consider the case 'prediction_type' = 'sample' (predict x0). min-SNR training strategy assign weights to each sample of the loss (as in the figure below, but replace $\gamma$ with values > $\gamma$:

This is quite counter-intuitive as (when t → 0), the denoising task of diffusion model is easier than (when t → T). Therefore, the weights (when t → T) should be higher than (when t → 0), and the formula for the weighs should be the inverse of min-SNR weighting, but the paper https://arxiv.org/abs/2303.09556 suggests the opposite weighting.

Can anyone please explain, please?

[AbrightWay]

@drhead could you please explain this phenomenon?

[drhead]

I wouldn't take the min-SNR weighting as being anything but an approximation of a good timestep weighting, it's meant to simplify the process and like anything that does that it's never going to be perfect or universally applicable. Even in the min-SNR paper they find that learning loss weights is a better way to achieve their objective of equalizing the effective loss weight of each timestep (and while they note tradeoffs I believe they are somewhat overstated, or are at least mitigated by using an MLP like what EDM2 uses).

If the part of this that's unexpected is that the curve is mirrored from what you expect I would recommend checking that the timesteps didn't get inverted somewhere, some implementations put the lowest SNR timestep on the opposite side.

[AbrightWay]

Thank you for your answer, @drhead ! However, I am still confused when it's common to put more weights the hard-to-optimize parts (prediction at high timesteps for prediction_type='sample'), but min-SNR do the opposite, i.e., weight those parts (prediction at high timesteps, e.g., t= 800) with extremely small value, to the order of 1e-4. This still bugs me. Could explain why min-SNR give such low weights to those hard-to-optimize tasks (prediction at high timesteps for prediction_type='sample')?
Thank you in advanced!