Unclear instruction on how to apply this framework to a new dataset. (e.g. get 2d / 3d pseudo poses)

[LadnerJonas]

Assuming I want to apply this framework to my dataset, where I have the calibration of my cameras and the video footage. Currently, no 2d or 3d pseudo poses exist.

What is the easiest way to get the 2d / 3d pseudo poses?

I have searched through the https://github.com/CAMMA-public/SelfPose3d/tree/main/pseudo_2d_labels_generation folder, but to my understanding, this would only generate 2d pseudo poses.

Given that the example files of the lib/dataset all use 3d poses, Is it possible to train with just HRNet-generated 2d poses without having to rewrite a big chunk of the code base?

[keqizero]

Hi,

Yes, the training of SelfPose3d only requires 2d pseudo poses, and we only provide the code to generate 2d pseudo poses. I am not sure what you mean by saying "the lib/dataset all use 3d poses".
Only during evaluation, we use the 3d GT poses.

[xinedison]

can you give a detail expression about how to train and evaluate this model on my own dataset. The readme for pseudo_2d_labels_generation is not clear for me.

[keqizero]

can you give a detail expression about how to train and evaluate this model on my own dataset. The readme for pseudo_2d_labels_generation is not clear for me.

To train on your own dataset, you need to:

1. generate 2d poses using an off-the-shelf pose estimator. The pseudo_2d_labels_generation is based on Detectron2. You can also use MMPose as well.
2. write the config file. You need to change the 3d space center and space size based on your own data. Please refer to issues.

For evaluation, you will need the 3d GT poses.

[xinedison]

Thank you for your reply. When I followed the step in pseudo_2d_labels_generation, the s1 generate image_info_train_panoptic.json from a TRAIN_DB_PATH, I want to know how can I generate the TRAIN_DB_PATH for my own dataset.

[keqizero]

Thank you for your reply. When I followed the step in pseudo_2d_labels_generation, the s1 generate image_info_train_panoptic.json from a TRAIN_DB_PATH, I want to know how can I generate the TRAIN_DB_PATH for my own dataset.

The pickle file (TRAIN_DB_PATH) is automatically generated when you run VoxelPose / SelfPose3d (see the _get_db function in lib/dataset), which contains the image names, camera parameters and the ground-truth joints (only for evaluation). You can write your own _get_db function following the same logic.

[xinedison]

Thanks for your reply. When I generate TRAIN_DB_PATH from lib/dataset/_get_db, I found the dataset read a pickle file group_train_cam5_pseudo_hrnet_hard_9videos.pkl for panoptic and pred_campus_maskrcnn_hrnet_coco.pkl for campus dataset. How can I generate the pickle file. Could you please share the script for generating this pickle db path. Thanks a lot.

[keqizero]

Thanks for your reply. When I generate TRAIN_DB_PATH from lib/dataset/_get_db, I found the dataset read a pickle file group_train_cam5_pseudo_hrnet_hard_9videos.pkl for panoptic and pred_campus_maskrcnn_hrnet_coco.pkl for campus dataset. How can I generate the pickle file. Could you please share the script for generating this pickle db path. Thanks a lot.

No, I don't think you need these files. For both VoxelPose and SelfPose3d, the code will first try to read if there is existing pkl file (as you listed). If there isn't, it will automatically generate a new pickle file by calling _get_db() function. Please check L116-L134 and L116-L135.