Private github repo fail for using log_model and log_metric

[sijin-dm]

Describe the bug

I have two version for using component. One is directly uploading local code to each pod, this work fine that we could see the models artifact and metric curve. The other one use the same code as the first one, except for using private github to load the code. In this case, the "dashboards", "artifacts" and "resources" pages show only "NO DATA", and "logs" page shows training output message normally.

To reproduce

1. Add private code connection

connections:
  - name: my-repo
    kind: git
    schema:
      url: https://github.com/xxx/my-repo
    secret:
      name: "github-secret-my-repo"

2. runing job config:

run:
  kind: job
  init:
    - connection: my-repo

3. How we use log_metric and log_model

# log metric
tracking.log_metric("val_loss", val_loss, step=epoch)
tracking.log_metric("val_precision", precision, step=epoch)
tracking.log_metric("val_recall", recall, step=epoch)

# log model
model_output_dir = tracking.get_outputs_path("models", is_dir=True)
ckpt_file = os.path.join(model_output_dir, 'checkpoint.pth.tar')
torch.save({xxx}, ckpt_file)
tracking.log_model(name="checkpoint", path=ckpt_file, framework="pytorch")

Expected behavior

Showing metric curve and saving models normally.

Environment

minikube: v1.15.1
polyaxon ce: 1.7.5

[polyaxon-team]

This is very hard to debug, you need to check if the git repo has some more information like a hard-coded NO_OP.

Also I would check if you added the local cache folder .polyaxon to your .gitignore and .dockerignore. If this folder was added to you git repo, then indeed the metrics and artifacts will be saved to a different run.

You can also validate that the run is running with the correct run-uuid:

...
print(tracking.TRACKING_RUN.run_uuid)
...

If the run_uuid does not correspond to the currently running run, then the cache is bundled somewhere (git repo or docker image).

[sijin-dm]

I didn't add .polyaxon to my .gitignore, after fixed it, everything is ok now. Thanks!