Add more extensive tests checking equality with pycocotools

[probicheaux]

Hi, I help maintain the rf-detr repo . You have an open PR in this repo. In looking at tests in this Repo, we see that you check equality of results with pycocotools only with one example. We would like to see a larger test suite to ensure equality across a wider range of possible inputs. Would you be willing to provide either a test script or adding to your test suite here comparing the 2 methods using a full suite of model predictions on Coco for all task types?

Obviously, we would prefer that rf-detr and rf-detr-seg are used to generate the predictions jsons for objdet and instseg respectively, whereas we don't have a keypoints model ready to use, so a yolo-pose model could generate said predictions json.

We understand the value of faster evaluation and would like to use this repo, but the above test is a bit of a blocker for us. Otherwise we will do the above test when our team finds time to prioritize it.

Thanks for your PR and your work maintaining this repo!

cc: @Borda @isaacrob

[Borda]

We would like to see a larger test suite to ensure equality across a wider range of possible inputs.

That is great suggestion, could you please be a bit more specific what is your expectation?

• various cases
• larger batches
• inputs with very large amount of predictions
• anything else?

[MiXaiLL76]

Hello, I initially did this manually, and that's what formed my unit tests.
Integration tests are indeed few and far between. I only tested a complete replacement on the yolo and mmdet frameworks using their tests, seeing identical results in my library.

Also, people who were integrating the library for their own work (on custom datasets) submitted this issue. The results for pycocotools and faster_coco_eval were identical, only the performance was usually different.

pycocotools -> slow and low memory usage
faster_coco_eval -> fast and higher memory usage.

There was also an issue #46, but the results were never implemented. These bugs exist in both pycocotools and mine. Everyone knows about them, everyone has "resigned themselves to them."

Overall, the tests presented at #71
are viable, but they need to be checked in more detail.

[MiXaiLL76]

We would like to see a larger test suite to ensure equality across a wider range of possible inputs.

That is great suggestion, could you please be a bit more specific what is your expectation?

* various cases

* larger batches

* inputs with very large amount of predictions

* anything else?

I was thinking about adding support for testing on the Coco dataset, but I realize it's a very time-consuming process (requiring downloading the dataset for testing) and I'm not sure it's needed here.

In any case, faster_coco_eval has much more extensive testing coverage than pycocotools.