Add Qwen3 Reranker model

[ayush1298]

closes #3718
Added 3 models:

1. Qwen3-Reranker-0.6B
2. Qwen3-Reranker-4B
3. Qwen3-Reranker-8B

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.
• The model is public, i.e., is available either as an API or the weights are publicly available to download

[Copilot]

Pull request overview

This PR adds support for three Qwen3 Reranker models (0.6B, 4B, and 8B variants) to the MTEB framework. The models are reranker models based on Qwen3 that can be used for relevance scoring tasks.

Changes:

• Added Qwen3RerankerWrapper class to load and run Qwen3 reranker models using causal language modeling with yes/no token probability scoring
• Added three ModelMeta configurations for Qwen3-Reranker-0.6B, Qwen3-Reranker-4B, and Qwen3-Reranker-8B
• Imported ScoringFunction from model_meta module to support metadata configuration

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Samoed]

Suggested change

	adapted_from=None,
	adapted_from="Qwen/Qwen3-4B",

[ayush1298]

Why this?

[Samoed]

This shouldn't be passed to model initialization

Suggested change

	torch_dtype=torch.float32,
	attn_implementation: str | None = None,
	batch_size: int = 32,
	max_length: int = 8192,

[ayush1298]

we can keep attn_implementation in init, right? will remove rest

[Samoed]

I don't think that instruction should be hardcoded

[ayush1298]

These is given on their hf page.

[Samoed]

Can you get task specific prompt? Instruction from batch will be only instruction retrieval/reranking tasks

[Samoed]

By the way you cat get implementation from their repo https://github.com/QwenLM/Qwen3-Embedding/blob/main/evaluation/qwen3_reranker_model.py

[ayush1298]

By the way you cat get implementation from their repo https://github.com/QwenLM/Qwen3-Embedding/blob/main/evaluation/qwen3_reranker_model.py

It's almost the same, but it uses vLLM. Should I use it?

[Samoed]

I don't think you need to change to vllm. I think it's better to use transformers or sentence transforemsrs, but I think this model is not compatible with sentence transformers

[ayush1298]

I don't think you need to change to vllm. I think it's better to use transformers or sentence transforemsrs, but I think this model is not compatible with sentence transformers

Their script on github is using vllm, and yes its not compatible with Sentence Transformers, so I think we can keep the current transformers implementation.

[ayush1298]

@Samoed I was able to run this code perfectly. Just 1 doubt is, for evaluation they have given these in their modelcard:

Evaluation results for reranking models. We use the retrieval subsets of MTEB(eng, v2), MTEB(cmn, v1), MMTEB and MTEB (Code), which are MTEB-R, CMTEB-R, MMTEB-R and MTEB-Code.

So, how can do evaluation on retrival subset only?

[Samoed]

We have example in docs https://embeddings-benchmark.github.io/mteb/usage/selecting_tasks/#filtering-benchmark-tasks

[ayush1298]

We have example in docs https://embeddings-benchmark.github.io/mteb/usage/selecting_tasks/#filtering-benchmark-tasks

import mteb
model_name="Qwen/Qwen3-Reranker-4B"
revision="f16fc5d5d2b9b1d0db8280929242745d79794ef5"
model = mteb.get_model(model_name)
benchmark = mteb.get_benchmark("MTEB(Code, v1)")

# Filter to only retrieval tasks
retrieval_tasks = mteb.filter_tasks(benchmark, task_types=["Retrieval"])
print(f"Found {len(retrieval_tasks)} retrieval tasks")
results = mteb.evaluate(model, tasks=retrieval_tasks)

Getting these, when trying to running above code, is it again because we allow only retrieval and not reranking one?
ValueError: CrossEncoder search requires top_ranked documents for reranking.

[Samoed]

Yes that's right. I think you can try to evaluate on their script on some reranking tasks and after that check your implementation

[ayush1298]

Yes that's right. I think you can try to evaluate on their script on some reranking tasks and after that check your implementation

Their script are using retrieval results in reranking.

Evaluate reranking models section in readme.md

[Samoed]

I think you can still run reranking tasks

[ayush1298]

I think you can still run reranking tasks

I am not able to run their code, getting an error because of I think conflict in dependencies.

[ayush1298]

@Samoed Could you try running it if possible? I tried it again. But not able to run it fully.