S3DIS Evalution Problem: AP=0 with pretrained model #70
Description
Hello, thank you for your great work!
I followed the steps in the README to evaluate the pretrained model on S3DIS (Area 5), but the results show almost all AP values as 0.
I downloaded the pretrained model and ran the following command python3 tools/test.py configs/s3dis/isbnet_s3dis_area5.yaml dataset/s3dis/head_s3dis_area5.pth. Here is the console output:
INFO - Load state dict from dataset/s3dis/head_s3dis_area5.pth
INFO - unexpected key in source state_dict: criterion.empty_weight
INFO - Load test dataset: 68 scans
INFO - Infer scene 1/68
INFO - Infer scene 11/68
INFO - Infer scene 21/68
INFO - Infer scene 31/68
INFO - Infer scene 41/68
INFO - Infer scene 51/68
INFO - Infer scene 61/68
INFO - Evaluate instance segmentation
################################################################
what : AP AP_50% AP_25% AR RC_50% RC_25%
################################################################
ceiling : 0.000 0.000 0.000 0.000 0.000 0.000
floor : 0.097 0.167 0.390 0.163 0.265 0.515
wall : 0.000 0.000 0.001 0.000 0.000 0.015
beam : 0.000 0.000 0.000 0.000 0.000 0.000
column : 0.000 0.000 0.000 0.000 0.000 0.000
window : 0.000 0.000 0.000 0.000 0.000 0.000
door : 0.000 0.000 0.000 0.000 0.000 0.000
chair : 0.000 0.000 0.000 0.000 0.000 0.000
table : 0.000 0.000 0.000 0.000 0.000 0.000
bookcase : 0.000 0.000 0.000 0.000 0.000 0.000
sofa : 0.000 0.000 0.000 0.000 0.000 0.000
board : 0.000 0.000 0.000 0.000 0.000 0.000
clutter : 0.000 0.000 0.000 0.000 0.000 0.000
----------------------------------------------------------------
average : 0.007 0.013 0.030 0.013 0.020 0.041
################################################################
INFO - Evaluate instance segmentation by S3DIS metrics
ISBNet/isbnet/evaluation/s3dis_eval.py:186: RuntimeWarning: invalid value encountered in double_scalars
prec = tp / (tp + fp)
################################################
what : MUCov MWCov Prec Rec
################################################
ceiling : 0.000 0.000 0.000 0.000
floor : 0.343 0.343 0.265 0.265
wall : 0.000 0.000 0.000 0.000
beam : 0.000 0.000 0.000 0.000
column : 0.000 0.000 nan 0.000
window : 0.000 0.000 0.000 0.000
door : 0.000 0.000 nan 0.000
chair : 0.001 0.001 0.000 0.000
table : 0.000 0.000 nan 0.000
bookcase : 0.000 0.000 0.000 0.000
sofa : 0.032 0.041 0.000 0.000
board : 0.000 0.000 nan 0.000
clutter : 0.001 0.001 0.000 0.000
################################################
mMUCov: 0.029013030115538803
mMWCov: 0.02975100740393172
mPrecision: 0.029411764705882353
mRecall: 0.020361990950226245
Would appreciate any insights on why the AP values are 0. Thanks!