Document image retrieval given a text query

[abdalimran]

Thanks for the gguf version of the model.
Can I use it for ViDoRe: The Visual Document Retrieval given a text query?
If so, can you please share a code snippet to find top_k within a list of images give na text query?

Thanks in advance!

[Andrei997]

Just to make sure I understand, are you trying to evaluate on ViDoRe tasks or are you trying to implement some kind of retrieval system? Regardless, for multi-modal embeddings you will need to use our fork of llama.cpp since originally image embeddings are not supported by llama.cpp.

Our fork is here: https://github.com/jina-ai/llama.cpp
You check the README here for more info: https://github.com/jina-ai/llama.cpp/tree/master/jina_embeddings

You can directly evaluate on ViDoRe tasks using our provided scripts (there are examples in the README). You can also inherit from the example python model class we provide for building retrieval systems if that is what you're looking to do.

We also have a blogpost about multimodal embeddings in llama.cpp: Multimodal Embeddings in Llama.cpp and GGUF