high variance in finger-spin

[stablegradients]

I am observing a high variability in results across different seeds for finger spin. Is there a particular reason for this?
What changes do we need to make to this repo to have an accurate reproduction of dreamer w/ the correct std. deviations.

[seolhokim]

Hello! Sorry for the late reply, and thanks for bringing this up!

I also noticed during my experiments that the results are highly sensitive to the random seed. If you use different seeds or make small changes in implementation details, it can lead to noticeable differences in performance—especially since we simplified a few techniques or skipped some parts to make the official implementation easier to use.

The first thing I recommend is to double-check whether you're using the same hyperparameters as in the paper. Small differences in settings like the imagination horizon, discount factor, lambda, learning rate, and action repeat can significantly affect the results.

As far as I know, the Dreamer paper does not explicitly report standard deviation values for Dreamer alone. In several experiments—including the Dreamer + representation learning results and other benchmarks—the plots show shaded areas that represent variability. However, they don’t specify whether these represent standard deviation or standard error, and there are no explicit numerical std values reported in the paper for Dreamer by itself.

If you could clarify which "std" you're referring to, we could discuss it further and try to figure it out together.

[stablegradients]

Hii
Yes, I am using the default hyper parameters as available in the config file which I believe equal the hyper parameters mentioned by the original paper.
the std. that I am referring to is the variation in evaluation score for different seeds, lets say finger spin, for different seeds I get eval score converging to 450, 600, 900 all in equal proportion.

Could you mention the parts that are not implemented in this or is different vs what was actually used. I see that you mentioned latent overshooting and n-step lambda value calculation but I believe latent overshooting was used as mentioned is App A Learning Updates Paragraph. Also Is the returns calculation very different in implementation? I fed them to LMs vs danijar hafners origianl tf code and the turn out to be the same. I could be wrong on this one.

[stablegradients]

I think the variance might be coming from the difference in the exploration strategy, tensorflow implements and exploration decay

[seolhokim]

the std. that I am referring to is the variation in evaluation score for different seeds, lets say finger spin, for different seeds I get eval score converging to 450, 600, 900 all in equal proportion.

I’m pretty sure they also observed variance across different seeds, even if it wasn’t explicitly reported. And even when using the same parameters, the internal workings of TensorFlow and PyTorch can introduce differences in behavior.
By the way, in your runs with the official Dreamer implementation, how much standard deviation do you typically see across seeds? Even if we’re using the same seeds between TensorFlow and PyTorch, I’d expect the results to differ a bit.

Could you mention the parts that are not implemented in this or is different vs what was actually used. I see that you mentioned latent overshooting and n-step lambda value calculation but I believe latent overshooting was used as mentioned is App A Learning Updates Paragraph. Also Is the returns calculation very different in implementation? I fed them to LMs vs danijar hafners origianl tf code and the turn out to be the same. I could be wrong on this one.

Good point! And first off, thanks for running those experiments. I haven’t tested it thoroughly myself, but if your returns match those from the original TensorFlow code, then it sounds like they’re doing the same thing there.

I think the variance might be coming from the difference in the exploration strategy, tensorflow implements and exploration decay

Yeah, that makes sense. At the end of the day, the variance could just come down to differences between the frameworks themselves. On top of that, things like learning rates (you probably matched them with the original implementation, but in my code, they’re different from the original), exploration noise (which I haven’t implemented at all in my YAML folder—though I’m not sure if this is what you’re referring to), neural net initialization, or even subtle differences in internal computation could all lead to some variation.

Thank you for the clarification!

[stablegradients]

I have not run on official tensorflow version due to cuda version restrictions on my lab's server.
I have seen these reported in this paper : https://arxiv.org/pdf/2004.04136

Here they report std dev for finger spin. and using fixed gaussian exploration matches with their mean and std dev for finger spin

My results:

[stablegradients]

This discussion should be helpful: https://claude.ai/share/5e55ee9e-a763-4286-85e8-5706e1ec7d2c

[seolhokim]

Good! :)