generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 143
Open
Description
Hello,
I just encountered an issue inside the training script.
Every training run creates two sibling experiment directories under logs/:
logs/project/XXX/ ← weights, config, etc.
logs/project/XXX/ ← renderings_training/ (empty)
get_timestamp() is called twice independently:
train_agent.py:182— generates the timestamp used for logging, config, checkpointsbase_task.py:82— generates a second timestamp a few seconds later, used for the simulator/video recorder'sexperiment_dir
Since the environment setup (IsaacGym init, wandb init, etc.) takes a few seconds between the two calls, the timestamps differ, creating two directories.
The video recorder writes to the second directory (renderings_training/), but when wandb is enabled the local video files are cleaned up after upload, leaving it empty.
I have created a commit that should solve the issue, should I open a PR?
Thanks for the help!
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels