Inconsistent Action between Recorded Dataset (Debug) and Actual Simulation Environment

[rxng8]

Dear Mees,

I hope you are doing well.

I created this issue because I believe that there is mismatch between the observation of the simulation and the dataset.

What I did was:

1. Download the debug dataset
2. Initialize the simulation environment using get_env(dataset_path, show_gui=False) in play_table_env.py
3. I acknowledge that there is scene mismatch between those two
4. For each loop body in the episode:

• I took the action from the demo from the dataset
• do env.step to get and observation
• save the image observation of the simulation as well as the image of the demo

Here is the result (left is the simulation and right is the actual demo image):

You can see that the action guide the robot in different direction, so there might be some mismatch in how the robot takes actions.
Also note that in this gif, we can see that the gripper probably works correctly.

Thank you very much for checking this out!
Best regards!

[clemgris]

This issue is related to the shape of the actions, which should be adjusted if you're using relative actions. You can refer to the wrapper at ./calvin/calvin_models/calvin_agent/wrappers/calvin_env_wrapper.py, where the step method automatically adapts the shape depending on whether the actions are relative or not.