Single-frame Benchmarking Page & Leaderboard

[mihaidusmanu]

With some collaborators, we are building a CodaBench Competition for an upcoming project & its related challenge. Since the new challenge uses the same metrics for the single-image/rig setup, we reused our work to create a LaMAR Competition too.

Link: https://www.codabench.org/competitions/7918

This PR:

• Adds links to benchmarking page (containing leaderboard).
• Adds script for merging results of different scenes / devices into a single zip file for submission.

TODO:

• Rerun and submit an initial baseline (Fusion + SP + LightGlue with default parameters).
• Make benchmarking page public.

TL;DR: Users need to be approved before submitting. 5 submissions / user / day. Leaderboard with overall recall @ thresholds & per-device split + detailed results. Main Leaderboard, Detailed submission results.

Note that the current competition does not support sequence metrics. Given CodaBench limitations (single leaderboard per competition, single phase running at a given time), the sequence evaluation should most likely be a different competition. In the meanwhile, we allow users to evaluate sequence methods and tag their submissions accordingly, but ask them to keep the submissions private.

[sarlinpe]

Amazing, thank you! One cannot see the leaderboard without being logged in?

[sarlinpe]

Users need to be approved before submitting.

How does the approval process look like? We automatically get a notification to @sympa.ethz.ch?

[mihaidusmanu]

Users need to be approved before submitting.

How does the approval process look like? We automatically get a notification to @sympa.ethz.ch?

Very good question 😄 I have to admit the docs are a bit fuzzy for CodaBench so it's not clear to me. I guess we'll see when the first person applies.

One cannot see the leaderboard without being logged in?

I was expecting it to be possible, but looking at other competitions and the docs, I didn't find a way yet. I'll open an issue and see if they have any suggestions.

[mihaidusmanu]

First person registered (thanks @hermannsblum) : I received an email on my CodaBench email (not the mailing list). If you have an account, let me know and I can add you as an admin / maybe create an organization.

Regarding public leaderboard, the CodaBench team replied and sadly that is not currently possible, but it was marked as an enhancement request so it might come (I'll try to read through their code when I have some time to get an idea of how hard that would be).