Skip to content

Training creates two experiment directories with different timestamps #44

@maedmatt

Description

@maedmatt

Hello,

I just encountered an issue inside the training script.

Every training run creates two sibling experiment directories under logs/:

logs/project/XXX/ ← weights, config, etc.
logs/project/XXX/ ← renderings_training/ (empty)

get_timestamp() is called twice independently:

  1. train_agent.py:182 — generates the timestamp used for logging, config, checkpoints
  2. base_task.py:82 — generates a second timestamp a few seconds later, used for the simulator/video recorder's experiment_dir

Since the environment setup (IsaacGym init, wandb init, etc.) takes a few seconds between the two calls, the timestamps differ, creating two directories.

The video recorder writes to the second directory (renderings_training/), but when wandb is enabled the local video files are cleaned up after upload, leaving it empty.

I have created a commit that should solve the issue, should I open a PR?

Thanks for the help!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions