Very weird artifcats or lips on finetuned model #83
Description
First of all thanks a lot for your amazing work in this field.
I am trying to finetune this on HDTF dataset and I can see very weird expression/artifcats. I have successfully run the preprocessing and everything. Any idea why such a problem occurs? I am training on 2xA100 GPU.
Sample result : https://drive.google.com/file/d/1gQMNLR8ZYAJKZJiF-dwg0LR2PUqe_SO0/view?usp=sharing
this is my training command
accelerate launch train.py \
--experiment_dir ~/workspace/train/experiments \
--experiment_name lipsync_finetune_hdtf_v2 \
--checkpoint ~/workspace/train/checkpoints/ditto_pytorch/models/lmdm_v0.4_hubert.pth \
--use_sc \
--use_last_frame \
--use_last_frame_loss \
--use_emo \
--use_eye_open \
--use_eye_ball \
--audio_feat_dim 1103 \
--motion_feat_dim 265 \
--batch_size 512 \
--num_workers 16 \
--epochs 50 \
--lr 1e-7 \
--save_ckpt_freq 10 \
--data_list_json ~/workspace/train/training_data/data_list.json \
--data_preload \
--data_preload_pkl ~/workspace/train/training_data/data_preload.pkl \
--mtn_mean_var_npy ~/workspace/train/training_data/data_preload_mtn_mean_var.npy
This is how my loss looks like
Epoch: 1, Global_Steps: 842, | scale_P: 0.081015 | yaw_P: 0.082173 | pitch_L: 0.036069 | t_L: 0.026452 | pitch_P: 0.086110 | roll_P: 0.060727 | pitch_A: 0.019594 | total_loss: 1.281273 | scale_L: 0.048095 | yaw_L: 0.016946 | yaw_A: 0.019805 | roll_V: 0.006391 | t_A: 0.010501 | scale_V: 0.007099 | exp_L: 0.190191 | yaw_V: 0.009318 | scale_A: 0.014669 | exp_A: 0.095632 | t_P: 0.113686 | roll_A: 0.013880 | exp_P: 0.270156 | pitch_V: 0.008835 | roll_L: 0.013408 | t_V: 0.006111 | exp_V: 0.044411 |
Epoch: 2, Global_Steps: 1684, | scale_P: 0.030388 | yaw_P: 0.030064 | pitch_L: 0.004180 | t_L: 0.005207 | pitch_P: 0.026580 | roll_P: 0.024826 | pitch_A: 0.014566 | total_loss: 0.631637 | scale_L: 0.010085 | yaw_L: 0.005684 | yaw_A: 0.014377 | roll_V: 0.005272 | t_A: 0.009131 | scale_V: 0.006038 | exp_L: 0.088804 | yaw_V: 0.006558 | scale_A: 0.012703 | exp_A: 0.075612 | t_P: 0.060662 | roll_A: 0.011704 | exp_P: 0.140672 | pitch_V: 0.006602 | roll_L: 0.003189 | t_V: 0.005185 | exp_V: 0.033547 |
Epoch: 3, Global_Steps: 2526, | scale_P: 0.025213 | yaw_P: 0.025921 | pitch_L: 0.002604 | t_L: 0.003777 | pitch_P: 0.022389 | roll_P: 0.021629 | pitch_A: 0.012548 | total_loss: 0.525497 | scale_L: 0.006917 | yaw_L: 0.003166 | yaw_A: 0.013117 | roll_V: 0.004934 | t_A: 0.008304 | scale_V: 0.005620 | exp_L: 0.060122 | yaw_V: 0.006005 | scale_A: 0.011951 | exp_A: 0.070006 | t_P: 0.052634 | roll_A: 0.010942 | exp_P: 0.114046 | pitch_V: 0.005825 | roll_L: 0.002320 | t_V: 0.004740 | exp_V: 0.030767 |
Epoch: 4, Global_Steps: 3368, | scale_P: 0.023235 | yaw_P: 0.024380 | pitch_L: 0.002145 | t_L: 0.003153 | pitch_P: 0.021052 | roll_P: 0.020306 | pitch_A: 0.011718 | total_loss: 0.482666 | scale_L: 0.005448 | yaw_L: 0.002639 | yaw_A: 0.012447 | roll_V: 0.004750 | t_A: 0.007857 | scale_V: 0.005360 | exp_L: 0.050756 | yaw_V: 0.005747 | scale_A: 0.011455 | exp_A: 0.065574 | t_P: 0.048634 | roll_A: 0.010525 | exp_P: 0.104644 | pitch_V: 0.005490 | roll_L: 0.001933 | t_V: 0.004495 | exp_V: 0.028923 |
Epoch: 5, Global_Steps: 4210, | scale_P: 0.021173 | yaw_P: 0.022636 | pitch_L: 0.001874 | t_L: 0.002814 | pitch_P: 0.019599 | roll_P: 0.019050 | pitch_A: 0.011234 | total_loss: 0.444364 | scale_L: 0.004505 | yaw_L: 0.002064 | yaw_A: 0.011837 | roll_V: 0.004602 | t_A: 0.007635 | scale_V: 0.005170 | exp_L: 0.041130 | yaw_V: 0.005479 | scale_A: 0.011114 | exp_A: 0.062633 | t_P: 0.045814 | roll_A: 0.010194 | exp_P: 0.094992 | pitch_V: 0.005289 | roll_L: 0.001703 | t_V: 0.004359 | exp_V: 0.027467 |
Epoch: 6, Global_Steps: 5052, | scale_P: 0.019912 | yaw_P: 0.021483 | pitch_L: 0.001497 | t_L: 0.002507 | pitch_P: 0.018624 | roll_P: 0.018322 | pitch_A: 0.010884 | total_loss: 0.415300 | scale_L: 0.003917 | yaw_L: 0.001761 | yaw_A: 0.011526 | roll_V: 0.004479 | t_A: 0.007312 | scale_V: 0.005002 | exp_L: 0.034782 | yaw_V: 0.005346 | scale_A: 0.010789 | exp_A: 0.059850 | t_P: 0.043316 | roll_A: 0.009908 | exp_P: 0.087012 | pitch_V: 0.005133 | roll_L: 0.001467 | t_V: 0.004192 | exp_V: 0.026281 |
Epoch: 7, Global_Steps: 5894, | scale_P: 0.019059 | yaw_P: 0.021027 | pitch_L: 0.001417 | t_L: 0.002326 | pitch_P: 0.018133 | roll_P: 0.017841 | pitch_A: 0.010650 | total_loss: 0.399786 | scale_L: 0.003461 | yaw_L: 0.001599 | yaw_A: 0.011252 | roll_V: 0.004399 | t_A: 0.007153 | scale_V: 0.004882 | exp_L: 0.032240 | yaw_V: 0.005227 | scale_A: 0.010572 | exp_A: 0.058460 | t_P: 0.041425 | roll_A: 0.009734 | exp_P: 0.082756 | pitch_V: 0.005028 | roll_L: 0.001471 | t_V: 0.004089 | exp_V: 0.025583 |
Epoch: 8, Global_Steps: 6736, | scale_P: 0.017871 | yaw_P: 0.020323 | pitch_L: 0.001372 | t_L: 0.002168 | pitch_P: 0.017330 | roll_P: 0.017294 | pitch_A: 0.010401 | total_loss: 0.380311 | scale_L: 0.003061 | yaw_L: 0.001778 | yaw_A: 0.010995 | roll_V: 0.004319 | t_A: 0.006962 | scale_V: 0.004760 | exp_L: 0.031658 | yaw_V: 0.005132 | scale_A: 0.010335 | exp_A: 0.053934 | t_P: 0.039548 | roll_A: 0.009535 | exp_P: 0.077262 | pitch_V: 0.004918 | roll_L: 0.001396 | t_V: 0.003973 | exp_V: 0.023986 |
Epoch: 9, Global_Steps: 7578, | scale_P: 0.017112 | yaw_P: 0.019747 | pitch_L: 0.001199 | t_L: 0.002037 | pitch_P: 0.016806 | roll_P: 0.016587 | pitch_A: 0.010147 | total_loss: 0.356230 | scale_L: 0.002739 | yaw_L: 0.001525 | yaw_A: 0.010716 | roll_V: 0.004200 | t_A: 0.006722 | scale_V: 0.004612 | exp_L: 0.027511 | yaw_V: 0.005004 | scale_A: 0.010059 | exp_A: 0.048507 | t_P: 0.037762 | roll_A: 0.009271 | exp_P: 0.072293 | pitch_V: 0.004801 | roll_L: 0.001255 | t_V: 0.003841 | exp_V: 0.021780 |
Epoch: 10, Global_Steps: 8420, | scale_P: 0.016206 | yaw_P: 0.019070 | pitch_L: 0.001209 | t_L: 0.001867 | pitch_P: 0.016116 | roll_P: 0.015905 | pitch_A: 0.010024 | total_loss: 0.346864 | scale_L: 0.002582 | yaw_L: 0.001436 | yaw_A: 0.010677 | roll_V: 0.004155 | t_A: 0.006677 | scale_V: 0.004553 | exp_L: 0.026646 | yaw_V: 0.004994 | scale_A: 0.009957 | exp_A: 0.048013 | t_P: 0.035914 | roll_A: 0.009172 | exp_P: 0.070296 | pitch_V: 0.004744 | roll_L: 0.001202 | t_V: 0.003792 | exp_V: 0.021659 |
Epoch: 11, Global_Steps: 9262, | scale_P: 0.015773 | yaw_P: 0.019038 | pitch_L: 0.001140 | t_L: 0.001813 | pitch_P: 0.016104 | roll_P: 0.015920 | pitch_A: 0.009846 | total_loss: 0.332729 | scale_L: 0.002318 | yaw_L: 0.001256 | yaw_A: 0.010379 | roll_V: 0.004094 | t_A: 0.006428 | scale_V: 0.004417 | exp_L: 0.024530 | yaw_V: 0.004866 | scale_A: 0.009665 | exp_A: 0.044676 | t_P: 0.034845 | roll_A: 0.009020 | exp_P: 0.066818 | pitch_V: 0.004667 | roll_L: 0.001176 | t_V: 0.003671 | exp_V: 0.020270 |
Epoch: 12, Global_Steps: 10104, | scale_P: 0.015210 | yaw_P: 0.018611 | pitch_L: 0.001052 | t_L: 0.001719 | pitch_P: 0.015783 | roll_P: 0.015971 | pitch_A: 0.009770 | total_loss: 0.326026 | scale_L: 0.002203 | yaw_L: 0.001242 | yaw_A: 0.010345 | roll_V: 0.004074 | t_A: 0.006423 | scale_V: 0.004348 | exp_L: 0.023246 | yaw_V: 0.004839 | scale_A: 0.009538 | exp_A: 0.044890 | t_P: 0.033910 | roll_A: 0.008967 | exp_P: 0.064289 | pitch_V: 0.004632 | roll_L: 0.001119 | t_V: 0.003651 | exp_V: 0.020192 |
Epoch: 13, Global_Steps: 10946, | scale_P: 0.014417 | yaw_P: 0.017695 | pitch_L: 0.001032 | t_L: 0.001607 | pitch_P: 0.015104 | roll_P: 0.014935 | pitch_A: 0.009668 | total_loss: 0.314065 | scale_L: 0.002027 | yaw_L: 0.001199 | yaw_A: 0.010225 | roll_V: 0.004015 | t_A: 0.006312 | scale_V: 0.004255 | exp_L: 0.021389 | yaw_V: 0.004778 | scale_A: 0.009346 | exp_A: 0.044041 | t_P: 0.032347 | roll_A: 0.008856 | exp_P: 0.061867 | pitch_V: 0.004578 | roll_L: 0.001068 | t_V: 0.003579 | exp_V: 0.019726 |
Epoch: 14, Global_Steps: 11788, | scale_P: 0.013778 | yaw_P: 0.017482 | pitch_L: 0.000982 | t_L: 0.001564 | pitch_P: 0.014679 | roll_P: 0.014700 | pitch_A: 0.009520 | total_loss: 0.304582 | scale_L: 0.001911 | yaw_L: 0.001094 | yaw_A: 0.010115 | roll_V: 0.003961 | t_A: 0.006163 | scale_V: 0.004192 | exp_L: 0.019904 | yaw_V: 0.004735 | scale_A: 0.009233 | exp_A: 0.042989 | t_P: 0.031010 | roll_A: 0.008725 | exp_P: 0.059485 | pitch_V: 0.004514 | roll_L: 0.001016 | t_V: 0.003497 | exp_V: 0.019334 |
Epoch: 15, Global_Steps: 12630, | scale_P: 0.014075 | yaw_P: 0.018084 | pitch_L: 0.000983 | t_L: 0.001540 | pitch_P: 0.015120 | roll_P: 0.015274 | pitch_A: 0.009523 | total_loss: 0.306541 | scale_L: 0.001787 | yaw_L: 0.001167 | yaw_A: 0.010106 | roll_V: 0.003979 | t_A: 0.006163 | scale_V: 0.004175 | exp_L: 0.019644 | yaw_V: 0.004743 | scale_A: 0.009160 | exp_A: 0.042702 | t_P: 0.031576 | roll_A: 0.008744 | exp_P: 0.059694 | pitch_V: 0.004519 | roll_L: 0.001019 | t_V: 0.003509 | exp_V: 0.019253 |
Epoch: 16, Global_Steps: 13472, | scale_P: 0.013329 | yaw_P: 0.017024 | pitch_L: 0.000880 | t_L: 0.001420 | pitch_P: 0.014264 | roll_P: 0.014365 | pitch_A: 0.009412 | total_loss: 0.293500 | scale_L: 0.001688 | yaw_L: 0.001030 | yaw_A: 0.010075 | roll_V: 0.003914 | t_A: 0.006102 | scale_V: 0.004112 | exp_L: 0.018264 | yaw_V: 0.004707 | scale_A: 0.009073 | exp_A: 0.041836 | t_P: 0.029123 | roll_A: 0.008632 | exp_P: 0.056558 | pitch_V: 0.004451 | roll_L: 0.000949 | t_V: 0.003445 | exp_V: 0.018848 |
Epoch: 17, Global_Steps: 14314, | scale_P: 0.013059 | yaw_P: 0.017004 | pitch_L: 0.000885 | t_L: 0.001426 | pitch_P: 0.014150 | roll_P: 0.014276 | pitch_A: 0.009360 | total_loss: 0.291778 | scale_L: 0.001606 | yaw_L: 0.001087 | yaw_A: 0.010051 | roll_V: 0.003892 | t_A: 0.006020 | scale_V: 0.004090 | exp_L: 0.017763 | yaw_V: 0.004682 | scale_A: 0.009035 | exp_A: 0.042282 | t_P: 0.028946 | roll_A: 0.008592 | exp_P: 0.055866 | pitch_V: 0.004425 | roll_L: 0.000950 | t_V: 0.003401 | exp_V: 0.018930 |
Epoch: 18, Global_Steps: 15156, | scale_P: 0.013076 | yaw_P: 0.017143 | pitch_L: 0.000912 | t_L: 0.001398 | pitch_P: 0.014345 | roll_P: 0.014453 | pitch_A: 0.009273 | total_loss: 0.287145 | scale_L: 0.001659 | yaw_L: 0.001079 | yaw_A: 0.009899 | roll_V: 0.003867 | t_A: 0.005954 | scale_V: 0.004028 | exp_L: 0.017164 | yaw_V: 0.004633 | scale_A: 0.008892 | exp_A: 0.040303 | t_P: 0.029062 | roll_A: 0.008511 | exp_P: 0.054593 | pitch_V: 0.004396 | roll_L: 0.000961 | t_V: 0.003375 | exp_V: 0.018168 |
Epoch: 19, Global_Steps: 15998, | scale_P: 0.012450 | yaw_P: 0.016661 | pitch_L: 0.000863 | t_L: 0.001361 | pitch_P: 0.013873 | roll_P: 0.014129 | pitch_A: 0.009229 | total_loss: 0.281079 | scale_L: 0.001493 | yaw_L: 0.001041 | yaw_A: 0.009738 | roll_V: 0.003847 | t_A: 0.005871 | scale_V: 0.003989 | exp_L: 0.017160 | yaw_V: 0.004589 | scale_A: 0.008815 | exp_A: 0.039324 | t_P: 0.027698 | roll_A: 0.008462 | exp_P: 0.053953 | pitch_V: 0.004376 | roll_L: 0.000908 | t_V: 0.003331 | exp_V: 0.017916 |
Epoch: 20, Global_Steps: 16840, | scale_P: 0.011957 | yaw_P: 0.016103 | pitch_L: 0.000816 | t_L: 0.001308 | pitch_P: 0.013279 | roll_P: 0.013629 | pitch_A: 0.009155 | total_loss: 0.273271 | scale_L: 0.001449 | yaw_L: 0.001008 | yaw_A: 0.009691 | roll_V: 0.003806 | t_A: 0.005828 | scale_V: 0.003925 | exp_L: 0.015987 | yaw_V: 0.004536 | scale_A: 0.008689 | exp_A: 0.038748 | t_P: 0.026702 | roll_A: 0.008393 | exp_P: 0.052278 | pitch_V: 0.004331 | roll_L: 0.000877 | t_V: 0.003290 | exp_V: 0.017486 |
Epoch: 21, Global_Steps: 17682, | scale_P: 0.011567 | yaw_P: 0.015798 | pitch_L: 0.000806 | t_L: 0.001256 | pitch_P: 0.013073 | roll_P: 0.013551 | pitch_A: 0.009119 | total_loss: 0.268482 | scale_L: 0.001385 | yaw_L: 0.000975 | yaw_A: 0.009672 | roll_V: 0.003792 | t_A: 0.005763 | scale_V: 0.003910 | exp_L: 0.015524 | yaw_V: 0.004531 | scale_A: 0.008676 | exp_A: 0.038572 | t_P: 0.025910 | roll_A: 0.008365 | exp_P: 0.050331 | pitch_V: 0.004305 | roll_L: 0.000890 | t_V: 0.003249 | exp_V: 0.017463 |
Epoch: 22, Global_Steps: 18524, | scale_P: 0.011552 | yaw_P: 0.015713 | pitch_L: 0.000804 | t_L: 0.001221 | pitch_P: 0.013094 | roll_P: 0.013637 | pitch_A: 0.009046 | total_loss: 0.265148 | scale_L: 0.001354 | yaw_L: 0.000947 | yaw_A: 0.009562 | roll_V: 0.003768 | t_A: 0.005720 | scale_V: 0.003871 | exp_L: 0.015628 | yaw_V: 0.004474 | scale_A: 0.008598 | exp_A: 0.037583 | t_P: 0.025415 | roll_A: 0.008299 | exp_P: 0.049453 | pitch_V: 0.004276 | roll_L: 0.000881 | t_V: 0.003231 | exp_V: 0.017019 |
Epoch: 23, Global_Steps: 19366, | scale_P: 0.011286 | yaw_P: 0.015791 | pitch_L: 0.000729 | t_L: 0.001187 | pitch_P: 0.012896 | roll_P: 0.013249 | pitch_A: 0.009013 | total_loss: 0.261385 | scale_L: 0.001232 | yaw_L: 0.000963 | yaw_A: 0.009596 | roll_V: 0.003757 | t_A: 0.005632 | scale_V: 0.003831 | exp_L: 0.014377 | yaw_V: 0.004494 | scale_A: 0.008519 | exp_A: 0.037089 | t_P: 0.024846 | roll_A: 0.008286 | exp_P: 0.049555 | pitch_V: 0.004258 | roll_L: 0.000816 | t_V: 0.003176 | exp_V: 0.016807 |
Epoch: 24, Global_Steps: 20208, | scale_P: 0.010846 | yaw_P: 0.015398 | pitch_L: 0.000740 | t_L: 0.001145 | pitch_P: 0.012674 | roll_P: 0.013215 | pitch_A: 0.008952 | total_loss: 0.257170 | scale_L: 0.001242 | yaw_L: 0.000910 | yaw_A: 0.009468 | roll_V: 0.003728 | t_A: 0.005620 | scale_V: 0.003798 | exp_L: 0.014473 | yaw_V: 0.004435 | scale_A: 0.008447 | exp_A: 0.036821 | t_P: 0.024095 | roll_A: 0.008216 | exp_P: 0.047988 | pitch_V: 0.004226 | roll_L: 0.000816 | t_V: 0.003159 | exp_V: 0.016760 |
Epoch: 25, Global_Steps: 21050, | scale_P: 0.010802 | yaw_P: 0.015426 | pitch_L: 0.000708 | t_L: 0.001134 | pitch_P: 0.012485 | roll_P: 0.013142 | pitch_A: 0.008933 | total_loss: 0.256251 | scale_L: 0.001205 | yaw_L: 0.000906 | yaw_A: 0.009609 | roll_V: 0.003717 | t_A: 0.005593 | scale_V: 0.003760 | exp_L: 0.014024 | yaw_V: 0.004503 | scale_A: 0.008361 | exp_A: 0.037006 | t_P: 0.023804 | roll_A: 0.008195 | exp_P: 0.047943 | pitch_V: 0.004213 | roll_L: 0.000796 | t_V: 0.003138 | exp_V: 0.016848 |
Epoch: 26, Global_Steps: 21892, | scale_P: 0.010543 | yaw_P: 0.014859 | pitch_L: 0.000721 | t_L: 0.001102 | pitch_P: 0.012144 | roll_P: 0.012741 | pitch_A: 0.008844 | total_loss: 0.250075 | scale_L: 0.001224 | yaw_L: 0.000900 | yaw_A: 0.009415 | roll_V: 0.003682 | t_A: 0.005508 | scale_V: 0.003731 | exp_L: 0.013528 | yaw_V: 0.004407 | scale_A: 0.008314 | exp_A: 0.036174 | t_P: 0.023139 | roll_A: 0.008120 | exp_P: 0.046436 | pitch_V: 0.004170 | roll_L: 0.000821 | t_V: 0.003098 | exp_V: 0.016455 |
Epoch: 27, Global_Steps: 22734, | scale_P: 0.010617 | yaw_P: 0.015063 | pitch_L: 0.000669 | t_L: 0.001051 | pitch_P: 0.012286 | roll_P: 0.012868 | pitch_A: 0.008820 | total_loss: 0.247450 | scale_L: 0.001119 | yaw_L: 0.000901 | yaw_A: 0.009436 | roll_V: 0.003676 | t_A: 0.005424 | scale_V: 0.003683 | exp_L: 0.012488 | yaw_V: 0.004402 | scale_A: 0.008207 | exp_A: 0.035869 | t_P: 0.023040 | roll_A: 0.008104 | exp_P: 0.045559 | pitch_V: 0.004156 | roll_L: 0.000763 | t_V: 0.003049 | exp_V: 0.016197 |
Epoch: 28, Global_Steps: 23576, | scale_P: 0.010373 | yaw_P: 0.014861 | pitch_L: 0.000695 | t_L: 0.001053 | pitch_P: 0.012189 | roll_P: 0.012745 | pitch_A: 0.008760 | total_loss: 0.245387 | scale_L: 0.001090 | yaw_L: 0.000865 | yaw_A: 0.009278 | roll_V: 0.003648 | t_A: 0.005404 | scale_V: 0.003672 | exp_L: 0.013176 | yaw_V: 0.004345 | scale_A: 0.008203 | exp_A: 0.035313 | t_P: 0.022379 | roll_A: 0.008043 | exp_P: 0.045310 | pitch_V: 0.004131 | roll_L: 0.000762 | t_V: 0.003038 | exp_V: 0.016053 |
Epoch: 29, Global_Steps: 24418, | scale_P: 0.010082 | yaw_P: 0.014188 | pitch_L: 0.000652 | t_L: 0.001032 | pitch_P: 0.011806 | roll_P: 0.012378 | pitch_A: 0.008706 | total_loss: 0.236812 | scale_L: 0.001029 | yaw_L: 0.000790 | yaw_A: 0.009181 | roll_V: 0.003619 | t_A: 0.005369 | scale_V: 0.003634 | exp_L: 0.012059 | yaw_V: 0.004276 | scale_A: 0.008119 | exp_A: 0.033865 | t_P: 0.021884 | roll_A: 0.007991 | exp_P: 0.043041 | pitch_V: 0.004096 | roll_L: 0.000722 | t_V: 0.003013 | exp_V: 0.015278 |
Epoch: 30, Global_Steps: 25260, | scale_P: 0.010038 | yaw_P: 0.014236 | pitch_L: 0.000660 | t_L: 0.001002 | pitch_P: 0.011781 | roll_P: 0.012321 | pitch_A: 0.008680 | total_loss: 0.236938 | scale_L: 0.001051 | yaw_L: 0.000801 | yaw_A: 0.009253 | roll_V: 0.003611 | t_A: 0.005310 | scale_V: 0.003622 | exp_L: 0.011534 | yaw_V: 0.004317 | scale_A: 0.008098 | exp_A: 0.034225 | t_P: 0.021718 | roll_A: 0.007966 | exp_P: 0.043395 | pitch_V: 0.004086 | roll_L: 0.000714 | t_V: 0.002981 | exp_V: 0.015538 |
Epoch: 31, Global_Steps: 26102, | scale_P: 0.009759 | yaw_P: 0.014230 | pitch_L: 0.000635 | t_L: 0.001004 | pitch_P: 0.011616 | roll_P: 0.012339 | pitch_A: 0.008647 | total_loss: 0.235609 | scale_L: 0.000990 | yaw_L: 0.000800 | yaw_A: 0.009153 | roll_V: 0.003600 | t_A: 0.005260 | scale_V: 0.003574 | exp_L: 0.011744 | yaw_V: 0.004266 | scale_A: 0.007993 | exp_A: 0.034011 | t_P: 0.021246 | roll_A: 0.007955 | exp_P: 0.043642 | pitch_V: 0.004062 | roll_L: 0.000716 | t_V: 0.002952 | exp_V: 0.015415 |
Epoch: 32, Global_Steps: 26944, | scale_P: 0.009651 | yaw_P: 0.013903 | pitch_L: 0.000614 | t_L: 0.000953 | pitch_P: 0.011390 | roll_P: 0.012136 | pitch_A: 0.008628 | total_loss: 0.232858 | scale_L: 0.000988 | yaw_L: 0.000769 | yaw_A: 0.009154 | roll_V: 0.003597 | t_A: 0.005262 | scale_V: 0.003576 | exp_L: 0.011476 | yaw_V: 0.004283 | scale_A: 0.007999 | exp_A: 0.033719 | t_P: 0.020851 | roll_A: 0.007945 | exp_P: 0.042853 | pitch_V: 0.004053 | roll_L: 0.000701 | t_V: 0.002941 | exp_V: 0.015416 |
Epoch: 33, Global_Steps: 27786, | scale_P: 0.009708 | yaw_P: 0.014167 | pitch_L: 0.000621 | t_L: 0.000974 | pitch_P: 0.011575 | roll_P: 0.012336 | pitch_A: 0.008639 | total_loss: 0.234996 | scale_L: 0.001002 | yaw_L: 0.000756 | yaw_A: 0.009309 | roll_V: 0.003602 | t_A: 0.005266 | scale_V: 0.003547 | exp_L: 0.011325 | yaw_V: 0.004360 | scale_A: 0.007929 | exp_A: 0.034436 | t_P: 0.020793 | roll_A: 0.007944 | exp_P: 0.043212 | pitch_V: 0.004062 | roll_L: 0.000714 | t_V: 0.002949 | exp_V: 0.015769 |
Epoch: 34, Global_Steps: 28628, | scale_P: 0.009363 | yaw_P: 0.013609 | pitch_L: 0.000594 | t_L: 0.000947 | pitch_P: 0.011240 | roll_P: 0.011954 | pitch_A: 0.008572 | total_loss: 0.227411 | scale_L: 0.000941 | yaw_L: 0.000755 | yaw_A: 0.009097 | roll_V: 0.003561 | t_A: 0.005133 | scale_V: 0.003529 | exp_L: 0.010701 | yaw_V: 0.004248 | scale_A: 0.007902 | exp_A: 0.033165 | t_P: 0.020216 | roll_A: 0.007863 | exp_P: 0.041315 | pitch_V: 0.004023 | roll_L: 0.000679 | t_V: 0.002879 | exp_V: 0.015126 |
Epoch: 35, Global_Steps: 29470, | scale_P: 0.009283 | yaw_P: 0.013819 | pitch_L: 0.000607 | t_L: 0.000933 | pitch_P: 0.011244 | roll_P: 0.011872 | pitch_A: 0.008597 | total_loss: 0.231819 | scale_L: 0.000927 | yaw_L: 0.000781 | yaw_A: 0.009232 | roll_V: 0.003585 | t_A: 0.005238 | scale_V: 0.003539 | exp_L: 0.011215 | yaw_V: 0.004329 | scale_A: 0.007930 | exp_A: 0.034525 | t_P: 0.020006 | roll_A: 0.007915 | exp_P: 0.042781 | pitch_V: 0.004037 | roll_L: 0.000692 | t_V: 0.002920 | exp_V: 0.015814 |
Epoch: 36, Global_Steps: 30312, | scale_P: 0.009308 | yaw_P: 0.013694 | pitch_L: 0.000571 | t_L: 0.000905 | pitch_P: 0.011148 | roll_P: 0.011995 | pitch_A: 0.008565 | total_loss: 0.228150 | scale_L: 0.000924 | yaw_L: 0.000720 | yaw_A: 0.009230 | roll_V: 0.003562 | t_A: 0.005171 | scale_V: 0.003499 | exp_L: 0.010416 | yaw_V: 0.004307 | scale_A: 0.007844 | exp_A: 0.033973 | t_P: 0.019655 | roll_A: 0.007884 | exp_P: 0.041742 | pitch_V: 0.004010 | roll_L: 0.000655 | t_V: 0.002873 | exp_V: 0.015497 |
Epoch: 37, Global_Steps: 31154, | scale_P: 0.009103 | yaw_P: 0.013343 | pitch_L: 0.000576 | t_L: 0.000888 | pitch_P: 0.011048 | roll_P: 0.011825 | pitch_A: 0.008450 | total_loss: 0.220086 | scale_L: 0.000918 | yaw_L: 0.000733 | yaw_A: 0.008915 | roll_V: 0.003517 | t_A: 0.005048 | scale_V: 0.003450 | exp_L: 0.010075 | yaw_V: 0.004165 | scale_A: 0.007738 | exp_A: 0.031628 | t_P: 0.019390 | roll_A: 0.007771 | exp_P: 0.039590 | pitch_V: 0.003963 | roll_L: 0.000655 | t_V: 0.002815 | exp_V: 0.014481 |
Epoch: 38, Global_Steps: 31996, | scale_P: 0.008844 | yaw_P: 0.013087 | pitch_L: 0.000562 | t_L: 0.000908 | pitch_P: 0.010721 | roll_P: 0.011529 | pitch_A: 0.008427 | total_loss: 0.218469 | scale_L: 0.000866 | yaw_L: 0.000695 | yaw_A: 0.008949 | roll_V: 0.003510 | t_A: 0.005021 | scale_V: 0.003410 | exp_L: 0.010381 | yaw_V: 0.004164 | scale_A: 0.007652 | exp_A: 0.031648 | t_P: 0.019065 | roll_A: 0.007766 | exp_P: 0.039395 | pitch_V: 0.003942 | roll_L: 0.000649 | t_V: 0.002798 | exp_V: 0.014477 |
Epoch: 39, Global_Steps: 32838, | scale_P: 0.008893 | yaw_P: 0.012893 | pitch_L: 0.000582 | t_L: 0.000860 | pitch_P: 0.010700 | roll_P: 0.011483 | pitch_A: 0.008411 | total_loss: 0.218628 | scale_L: 0.000869 | yaw_L: 0.000714 | yaw_A: 0.008922 | roll_V: 0.003499 | t_A: 0.005050 | scale_V: 0.003387 | exp_L: 0.010651 | yaw_V: 0.004145 | scale_A: 0.007591 | exp_A: 0.032121 | t_P: 0.018910 | roll_A: 0.007740 | exp_P: 0.039205 | pitch_V: 0.003934 | roll_L: 0.000658 | t_V: 0.002808 | exp_V: 0.014601 |
Epoch: 40, Global_Steps: 33680, | scale_P: 0.008772 | yaw_P: 0.012948 | pitch_L: 0.000543 | t_L: 0.000832 | pitch_P: 0.010742 | roll_P: 0.011588 | pitch_A: 0.008376 | total_loss: 0.215604 | scale_L: 0.000850 | yaw_L: 0.000670 | yaw_A: 0.008887 | roll_V: 0.003491 | t_A: 0.004984 | scale_V: 0.003347 | exp_L: 0.009581 | yaw_V: 0.004121 | scale_A: 0.007498 | exp_A: 0.031642 | t_P: 0.018673 | roll_A: 0.007718 | exp_P: 0.038701 | pitch_V: 0.003918 | roll_L: 0.000616 | t_V: 0.002771 | exp_V: 0.014334 |
Epoch: 41, Global_Steps: 34522, | scale_P: 0.008399 | yaw_P: 0.012455 | pitch_L: 0.000535 | t_L: 0.000855 | pitch_P: 0.010310 | roll_P: 0.011220 | pitch_A: 0.008345 | total_loss: 0.212253 | scale_L: 0.000842 | yaw_L: 0.000683 | yaw_A: 0.008841 | roll_V: 0.003472 | t_A: 0.004948 | scale_V: 0.003325 | exp_L: 0.009499 | yaw_V: 0.004108 | scale_A: 0.007436 | exp_A: 0.031300 | t_P: 0.018338 | roll_A: 0.007683 | exp_P: 0.038084 | pitch_V: 0.003899 | roll_L: 0.000613 | t_V: 0.002752 | exp_V: 0.014312 |
Epoch: 42, Global_Steps: 35364, | scale_P: 0.008534 | yaw_P: 0.012399 | pitch_L: 0.000545 | t_L: 0.000833 | pitch_P: 0.010263 | roll_P: 0.011054 | pitch_A: 0.008352 | total_loss: 0.211841 | scale_L: 0.000826 | yaw_L: 0.000701 | yaw_A: 0.008855 | roll_V: 0.003477 | t_A: 0.004942 | scale_V: 0.003321 | exp_L: 0.009494 | yaw_V: 0.004122 | scale_A: 0.007451 | exp_A: 0.031291 | t_P: 0.018105 | roll_A: 0.007702 | exp_P: 0.038009 | pitch_V: 0.003895 | roll_L: 0.000621 | t_V: 0.002739 | exp_V: 0.014313 |
Epoch: 43, Global_Steps: 36206, | scale_P: 0.008787 | yaw_P: 0.013281 | pitch_L: 0.000525 | t_L: 0.000818 | pitch_P: 0.010717 | roll_P: 0.011483 | pitch_A: 0.008363 | total_loss: 0.217923 | scale_L: 0.000817 | yaw_L: 0.000666 | yaw_A: 0.008964 | roll_V: 0.003480 | t_A: 0.004951 | scale_V: 0.003300 | exp_L: 0.009368 | yaw_V: 0.004178 | scale_A: 0.007387 | exp_A: 0.032302 | t_P: 0.018288 | roll_A: 0.007706 | exp_P: 0.040528 | pitch_V: 0.003904 | roll_L: 0.000603 | t_V: 0.002742 | exp_V: 0.014766 |
Epoch: 44, Global_Steps: 37048, | scale_P: 0.008280 | yaw_P: 0.012391 | pitch_L: 0.000535 | t_L: 0.000807 | pitch_P: 0.010247 | roll_P: 0.010956 | pitch_A: 0.008296 | total_loss: 0.209820 | scale_L: 0.000791 | yaw_L: 0.000678 | yaw_A: 0.008852 | roll_V: 0.003459 | t_A: 0.004881 | scale_V: 0.003283 | exp_L: 0.009395 | yaw_V: 0.004117 | scale_A: 0.007349 | exp_A: 0.030993 | t_P: 0.017891 | roll_A: 0.007661 | exp_P: 0.037552 | pitch_V: 0.003871 | roll_L: 0.000615 | t_V: 0.002704 | exp_V: 0.014217 |
Epoch: 45, Global_Steps: 37890, | scale_P: 0.008482 | yaw_P: 0.012480 | pitch_L: 0.000551 | t_L: 0.000814 | pitch_P: 0.010376 | roll_P: 0.011250 | pitch_A: 0.008305 | total_loss: 0.210686 | scale_L: 0.000806 | yaw_L: 0.000706 | yaw_A: 0.008811 | roll_V: 0.003465 | t_A: 0.004878 | scale_V: 0.003260 | exp_L: 0.009499 | yaw_V: 0.004104 | scale_A: 0.007277 | exp_A: 0.030891 | t_P: 0.018127 | roll_A: 0.007657 | exp_P: 0.037609 | pitch_V: 0.003880 | roll_L: 0.000617 | t_V: 0.002711 | exp_V: 0.014131 |
Epoch: 46, Global_Steps: 38732, | scale_P: 0.008141 | yaw_P: 0.012239 | pitch_L: 0.000517 | t_L: 0.000793 | pitch_P: 0.010088 | roll_P: 0.010940 | pitch_A: 0.008303 | total_loss: 0.209511 | scale_L: 0.000805 | yaw_L: 0.000664 | yaw_A: 0.008898 | roll_V: 0.003464 | t_A: 0.004877 | scale_V: 0.003237 | exp_L: 0.009098 | yaw_V: 0.004125 | scale_A: 0.007241 | exp_A: 0.031688 | t_P: 0.017760 | roll_A: 0.007677 | exp_P: 0.037339 | pitch_V: 0.003867 | roll_L: 0.000607 | t_V: 0.002699 | exp_V: 0.014445 |
Epoch: 47, Global_Steps: 39574, | scale_P: 0.008177 | yaw_P: 0.012205 | pitch_L: 0.000512 | t_L: 0.000776 | pitch_P: 0.010078 | roll_P: 0.010981 | pitch_A: 0.008211 | total_loss: 0.205128 | scale_L: 0.000753 | yaw_L: 0.000635 | yaw_A: 0.008727 | roll_V: 0.003424 | t_A: 0.004785 | scale_V: 0.003200 | exp_L: 0.008904 | yaw_V: 0.004042 | scale_A: 0.007159 | exp_A: 0.030325 | t_P: 0.017321 | roll_A: 0.007586 | exp_P: 0.036437 | pitch_V: 0.003822 | roll_L: 0.000594 | t_V: 0.002651 | exp_V: 0.013821 |
Epoch: 48, Global_Steps: 40416, | scale_P: 0.007986 | yaw_P: 0.011872 | pitch_L: 0.000511 | t_L: 0.000765 | pitch_P: 0.009961 | roll_P: 0.010722 | pitch_A: 0.008224 | total_loss: 0.204076 | scale_L: 0.000743 | yaw_L: 0.000645 | yaw_A: 0.008733 | roll_V: 0.003423 | t_A: 0.004820 | scale_V: 0.003183 | exp_L: 0.008754 | yaw_V: 0.004047 | scale_A: 0.007114 | exp_A: 0.030332 | t_P: 0.017319 | roll_A: 0.007589 | exp_P: 0.036391 | pitch_V: 0.003825 | roll_L: 0.000595 | t_V: 0.002662 | exp_V: 0.013858 |
Epoch: 49, Global_Steps: 41258, | scale_P: 0.008016 | yaw_P: 0.012054 | pitch_L: 0.000507 | t_L: 0.000762 | pitch_P: 0.010038 | roll_P: 0.010909 | pitch_A: 0.008225 | total_loss: 0.204793 | scale_L: 0.000739 | yaw_L: 0.000638 | yaw_A: 0.008818 | roll_V: 0.003429 | t_A: 0.004798 | scale_V: 0.003164 | exp_L: 0.008767 | yaw_V: 0.004070 | scale_A: 0.007077 | exp_A: 0.030616 | t_P: 0.017367 | roll_A: 0.007610 | exp_P: 0.036184 | pitch_V: 0.003823 | roll_L: 0.000589 | t_V: 0.002649 | exp_V: 0.013946 |
Epoch: 50, Global_Steps: 42100, | scale_P: 0.008217 | yaw_P: 0.012266 | pitch_L: 0.000518 | t_L: 0.000767 | pitch_P: 0.010140 | roll_P: 0.011098 | pitch_A: 0.008175 | total_loss: 0.204508 | scale_L: 0.000729 | yaw_L: 0.000624 | yaw_A: 0.008695 | roll_V: 0.003413 | t_A: 0.004752 | scale_V: 0.003132 | exp_L: 0.008601 | yaw_V: 0.004051 | scale_A: 0.006986 | exp_A: 0.030212 | t_P: 0.017528 | roll_A: 0.007553 | exp_P: 0.036064 | pitch_V: 0.003809 | roll_L: 0.000578 | t_V: 0.002630 | exp_V: 0.013970 |
I tried doing inference on most of the checkpoints including at epoch 1, 2, 5, 10, 20, 30, 40 and 50. I see such a behaviour even at train_1.pt checkpoint ( I say this to rule out any overfit since my learning rate is very low as well). Any suggestions would be helpful.