How to use reranking models (e.g., jina-reranker-v2) with Hugot? #75
-
|
Hi everyone! First of all, thank you for this wonderful project! I'm currently considering integrating it into one of my pet projects (a simple RAG service) and I have some questions about how to use certain models from Hugging Face. For example, how could I use a reranking model like jinaai/jina-reranker-v2-base-multilingual? I've run some tests but the results don't seem correct. Could anyone provide some guidance or pointers? Thanks in advance! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 12 replies
-
|
Hi there! Thank you for your interest. Short answer is this is currently not (yet) supported in that the FeatureExtractionPipeline is designed for (sentence) embedding, e.g. like all-miniLM from sentence transformers, not reranking (i.e. cross encoder) models like the Jina one (which rerank rather than producing embeddings). Implementing cross encoders for hugot should not be hard however, it amounts to basically converting the logic of the We had the request to use CrossEncoder models before in the past so it might be something I can take a look at, can't provide timelines though - might have some time this weekend |
Beta Was this translation helpful? Give feedback.
-
|
@Hoekz I believe I found the mismatch issue, as is usually the case it was in the tokenization which assumed BERT tokenization and added extra tokens. Can you try with this branch: https://github.com/knights-analytics/hugot/tree/cross-encoder-fix I get matching logits/ranking across both the Jina's and the Marco model. |
Beta Was this translation helpful? Give feedback.
-
|
I'm still seeing discrepancies on my side...for a JS example: import { env, AutoTokenizer, AutoModelForSequenceClassification } from '@xenova/transformers'
import os from 'os'
env.cacheDir = os.tmpdir()
const sigmoid = x => 1 / (1 + Math.exp(-x))
export async function rerank(query, documents) {
if (!documents.length) {
return []
}
const model = await AutoModelForSequenceClassification.from_pretrained('Xenova/ms-marco-MiniLM-L-6-v2')
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/ms-marco-MiniLM-L-6-v2')
const encoding = await tokenizer(documents.map(() => query), {
text_pair: documents.map(({ text }) => text),
padding: true,
truncation: true,
})
const embeddings = await model(encoding)
const scores = embeddings.logits.data.map(sigmoid)
const toSort = documents.map((doc, i) => ({ doc, score: scores[i] }))
return toSort.sort((a, b) => b.score - a.score).map(({ doc, score }) => ({ ...doc, score }))
}
await rerank('What is the capital of France?', [
{ text: 'The capital of France is Paris.' },
{ text: 'The capital of Germany is Berlin.' },
])
// [
// {
// text: 'The capital of France is Paris.',
// score: 0.9998955130577087
// },
// {
// text: 'The capital of Germany is Berlin.',
// score: 0.10640902072191238
// }
// ]Whereas when hitting my Go server: curl 'http://[::1]:8644/rerank' \
--request POST \
--header 'Content-Type: application/json' \
--data '{
"documents": [
"The capital of France is Paris.",
"The capital of Germany is Berlin."
],
"query": "What is the capital of France?"
}'
# {
# "$schema": "https://[::1]:8644/schemas/RankedDocuments.json",
# "documents": [
# {
# "document": "The capital of France is Paris.",
# "score": 0.1041819229722023
# },
# {
# "document": "The capital of Germany is Berlin.",
# "score": 0.013263199478387833
# }
# ]
# }The scores/logits are different this time around though, running the same query against {
"$schema": "https://[::1]:8644/schemas/RankedDocuments.json",
"documents": [
{
"document": "The capital of France is Paris.",
"score": 0.029932620003819466
},
{
"document": "The capital of Germany is Berlin.",
"score": 0.0033979716245085
}
]
} |
Beta Was this translation helpful? Give feedback.
-
|
@Hoekz allright, I missed another thing: BERT type models use a different pattern for the token type IDs, i.e. 0...0,1....1 for the two sequences. query := "Organic skincare products for sensitive skin" and I get from go: expectedBert := []Expected{ which is consistent with python: Scores (sigmoid): |
Beta Was this translation helpful? Give feedback.
-
|
That seems to have worked! Thanks so much for working on this. |
Beta Was this translation helpful? Give feedback.
-
|
great, will make a patch release for it |
Beta Was this translation helpful? Give feedback.
Hi there! Thank you for your interest.
Short answer is this is currently not (yet) supported in that the FeatureExtractionPipeline is designed for (sentence) embedding, e.g. like all-miniLM from sentence transformers, not reranking (i.e. cross encoder) models like the Jina one (which rerank rather than producing embeddings).
Implementing cross encoders for hugot should not be hard however, it amounts to basically converting the logic of the
AutoModelForSequenceClassificationhuggingface AutoModel to replicate what they do e.g. here https://huggingface.co/jinaai/jina-reranker-v1-tiny-en in the huggingface transformers approach, creating a new pipelineType like e.g.CrossEncoderPipeline.We…