nuScenes cannot reproduce the results #104
Description
Hi,
I tried to run the evaluation using the models which were removed in this commit. Then I ran the quantative analysis but the results are far worse than that in the paper.
I split the full data set into train, test and val using the provided process.py script following the guide
In case you also want a validation set generated (by default this will just produce the training and test sets), replace line 406 in process_data.py with:
val_scene_names = val_scenes
Below is the results for vehicle
FDE Results for: int_ee
-----------------PH: 2 -------------------
FDE Mean @1.0s: 0.7134760695972244
RB Viols @1.0s: 0.00662769225994662
FDE @1.0s: 0.5380502227267765
KDE @1.0s: -0.5028318785009163
----------------------------------------------
-----------------PH: 4 -------------------
FDE Mean @2.0s: 1.9975224575706407
RB Viols @2.0s: 0.034035627831916083
FDE @2.0s: 1.4918774864657154
KDE @2.0s: 0.418281733790496
----------------------------------------------
-----------------PH: 6 -------------------
FDE Mean @3.0s: 3.7033154076606456
RB Viols @3.0s: 0.08517236670597728
FDE @3.0s: 2.699203884395092
KDE @3.0s: 1.0535876062621374
----------------------------------------------
-----------------PH: 8 -------------------
FDE Mean @4.0s: 5.804041245565657
RB Viols @4.0s: 0.1444243994786171
FDE @4.0s: 4.179235735624605
KDE @4.0s: 1.5966484710987425
----------------------------------------------
FDE Results for: int_ee_me
-----------------PH: 2 -------------------
FDE Mean @1.0s: 0.7008344335547151
RB Viols @1.0s: 0.006146064800446899
FDE @1.0s: 0.5922799374447909
KDE @1.0s: -1.2356929101581047
----------------------------------------------
-----------------PH: 4 -------------------
FDE Mean @2.0s: 1.9075230766300963
RB Viols @2.0s: 0.026667106324871206
FDE @2.0s: 1.6498426923843936
KDE @2.0s: -0.15959527942645782
----------------------------------------------
-----------------PH: 6 -------------------
FDE Mean @3.0s: 3.51359302112881
RB Viols @3.0s: 0.07011046800322761
FDE @3.0s: 2.997894682140819
KDE @3.0s: 0.6260908819550914
----------------------------------------------
-----------------PH: 8 -------------------
FDE Mean @4.0s: 5.540642291109636
RB Viols @4.0s: 0.12788711129042268
FDE @4.0s: 4.693464571494656
KDE @4.0s: 1.2753486907234253
----------------------------------------------
FDE Results for: vel_ee
-----------------PH: 2 -------------------
FDE Mean @1.0s: 1.0865936039715742
RB Viols @1.0s: 0.008077260877661226
FDE @1.0s: 0.8246370557999223
KDE @1.0s: 5.759522830588884
----------------------------------------------
-----------------PH: 4 -------------------
FDE Mean @2.0s: 2.53460643671403
RB Viols @2.0s: 0.04206272112221464
FDE @2.0s: 1.8659767084663388
KDE @2.0s: 4.496077915865272
----------------------------------------------
-----------------PH: 6 -------------------
FDE Mean @3.0s: 4.303775744650805
RB Viols @3.0s: 0.09549359133511266
FDE @3.0s: 3.0955748259629154
KDE @3.0s: 4.258032260852793
----------------------------------------------
-----------------PH: 8 -------------------
FDE Mean @4.0s: 6.380693848054547
RB Viols @4.0s: 0.15273570852212773
FDE @4.0s: 4.513898783223498
KDE @4.0s: 4.306559906932642
----------------------------------------------
FDE Results for: robot
-----------------PH: 2 -------------------
FDE Mean @1.0s: 0.646239083876121
RB Viols @1.0s: 0.005744594448254702
FDE @1.0s: 0.586071305811502
KDE @1.0s: 1.6044233667942858
----------------------------------------------
-----------------PH: 4 -------------------
FDE Mean @2.0s: 1.787512170553925
RB Viols @2.0s: 0.023858047864852183
FDE @2.0s: 1.6299969047674239
KDE @2.0s: 2.676255090655095
----------------------------------------------
-----------------PH: 6 -------------------
FDE Mean @3.0s: 3.4118360922321065
RB Viols @3.0s: 0.05805988521098798
FDE @3.0s: 3.11633133756069
KDE @3.0s: 3.4138199082936795
----------------------------------------------
-----------------PH: 8 -------------------
FDE Mean @4.0s: 5.510224740288833
RB Viols @4.0s: 0.10027347218712775
FDE @4.0s: 5.040514456981584
KDE @4.0s: 3.969074325345989
----------------------------------------------
The commands for running the evaluation is shown below
#!/bin/bash
for folder in int_ee int_ee_me robot vel_ee; do
for ph in 2 4 6 8; do
cmd="python evaluate.py --model models/$folder --checkpoint=12 --data ../processed/nuScenes_test_full.pkl --output_path results --output_tag $folder --node_type VEHICLE --prediction_horizon $ph"
echo $cmd
eval $cmd
done
done
for folder in int_ee_me vel_ee; do
for ph in 2 4 6 8; do
cmd="python evaluate.py --model models/$folder --checkpoint=12 --data ../processed/nuScenes_test_full.pkl --output_path results --output_tag ${folder}_ped --node_type PEDESTRIAN --prediction_horizon $ph"
echo $cmd
eval $cmd
done
doneDid I do anything wrong for the evaluation or analysis? Thank you!