Questions about the code ‘similarities = -similarities’

[Syyabb]

Your work is excellent, but I don't understand the purpose of the line 'similarities = -similarities'. Could you please explain it to me?

[ChimpOnCloud]

same question

[jay-bo]

same question. May I ask how to obtain the evaluation results in the thesis?

[Danielement321]

I have conducted a script to compute the cosine similarity between the encoded text and each image token with GPT's help. To my surprise, the observation is somewhat contradictory to the authors', i.e., the image tokens with higher similarity visually have less similarity. The phenomena is confusing and not correspond to the paper.

[bbsngg]

Your work is excellent, but I don't understand the purpose of the line 'similarities = -similarities'. Could you please explain it to me?

Hello, thank you very much for your question. We actually discovered this issue in our initial experiments as well. When we visualized the results, we found that if we didn't add a negative sign when taking the feature scores, the conclusion was the opposite. We believe the possible reason is that the number of layers of the visual features we selected is different from the number of layers used when CLIP calculated its loss during training.