Halluc Metric

[MouxiaoHuang]

Sorry to bother again.

Does this file provides a metric to calculate the overall halluc metric: https://github.com/patrick-tssn/VideoHallucer/blob/main/evaluations/evaluation_halluc.py?

In the line 37 and line 38, which line should be used? From the paper, section 3.3 says We then calculate the overall accuracy by considering both the basic and hallucinated questions as a paired set, marking it as a hit only if both questions are answered correctly., thus should we use re.search(y_pattern, basic_pred, re.IGNORECASE) and re.search(n_pattern, halluc_pred, re.IGNORECASE) to calculate the overall metric? (if I understand correctly)

[MouxiaoHuang]

---

Let me be more specific.

For example, the Accuracy on VideoHallucer in the table,

Basic = model predicts basic yes / basic question number, which is a simple binary accuracy score

Hallucinated = model predicts hallucinated no / hallucinated question number, which is also a simple binary accuracy score

Overall = ((model predicts basic yes) AND (model predicts hallucinated no)) / (number of model predicts basic yes)

Do the calculation formulas right?

[patrick-tssn]

Let me be more specific.

For example, the Accuracy on VideoHallucer in the table,

Basic = model predicts basic yes / basic question number, which is a simple binary accuracy score

Hallucinated = model predicts hallucinated no / hallucinated question number, which is also a simple binary accuracy score

Overall = ((model predicts basic yes) AND (model predicts hallucinated no)) / (number of model predicts basic yes)

Do the calculation formulas right?

No, mostly right, but the denominator is wrong. Since the basic questions and hallucinated questions are paired—which means for each basic question, there is a hallucinated question—the number of basic questions is the same as the hallucinated questions. So the score is: Overall = ((model predicts basic yes) AND (model predicts hallucinated no)) / (number of basic questions or number of hallucinated questions).

[patrick-tssn]

For further discussion, you may add my WeChat Wang-jc1996