Please add a better example for embeddings using pure go

[devlux76]

I love what you've done here and I really want to use this library. You have so many examples that are close to what I need, but I'm really having difficulty figure out how to put these pieces into something coherent.

My use case...
I'd like to use sentence-transformers/all-MiniLM-L6-v2 and the onnx-gomlx library to...

Download the model from hf, transform the model to gomlx, create a tokenizer, tokenize a string, get the embeddings for the string.
The problem is that the example uses magic numbers you got out of python. Yet there is a tokenizer for this particular model.
How do we get from this...

// Execute it with GoMLX/XLA:
sentences := []string{
    "This is an example sentence",
    "Each sentence is converted"}
//... tokenize ...
inputIDs := [][]int64{
    {101, 2023, 2003, 2019, 2742, 6251,  102},
    { 101, 2169, 6251, 2003, 4991,  102,    0}}
tokenTypeIDs := [][]int64{
    {0, 0, 0, 0, 0, 0, 0},
    {0, 0, 0, 0, 0, 0, 0}}
attentionMask := [][]int64{
    {1, 1, 1, 1, 1, 1, 1},
    {1, 1, 1, 1, 1, 1, 0}}
...

To something more like...

// Load tokenizer
	tokenizer, err := tokenizers.New(repo)
	if err != nil {
		panic(fmt.Sprintf("failed to load tokenizer: %v", err))
	}
tokens := tokenizer.Encode(sentences)
embeddings, := model.GetEmbeddings(tokens);

An example like that would be a life saver and thanks so much again for all you do!

[janpfeifer]

hey @devlux76 , thanks for reaching out.

So the onnx-gomlx library aims to provide a somewhat lower level API, for folks manipulating models. One can for instance fine-tune an ONNX model using GoMLX and then save back to ONNX, or co-fine-tune two different instances of a model to train heterogeneous 2-tower models for indexing recommenders (e.g.: music, videos) or for RAGs, etc. All these require good knowledge of GoMLX.

onnx-gomlx also can be the engine (using GoMLX's XLA or a pure Go backend) used to serve a simple model, but it doesn't yet provide a "higher-level" API, as you described -- notice every model uses inputs and outputs in different ways (different input/output names, and formats for instance for masks), so such an API would also have to have specialized code per model.

A few things that can help you here:

• github.com/knights-analytics/hugot: It uses onnx-gomlx, and the API is higher level, closer to what you want. onnx-gomlx were developed jointly with them. (CC'ing folks)

• github.com/gomlx/go-huggingface: Tokenizers and downloading of models. It is also used by Hugot. I'm trying to add there the APIs for interacting with HuggingFace models, including tokenizers and sharing of cache with the python version.

This project is somewhat fresh: it is already being used for serious models in production, but it does not cover every ONNX model yet -- some ONNX ops are being implemented as needed. Suggestions (and contributions) are always welcome -- also in Hugot, I assume.

cheers

[RJKeevil]

Hi @devlux76, as @janpfeifer says, I think hugot might be great for what you are asking, we essentially take the goMLX backend and wrap it with tokenizers and huggingface pipeline logic so that you can go straight from e.g. strings to model output. Take a look at this example from our tests https://github.com/knights-analytics/hugot/blob/68dda1db9606c7f9cc9f93c07387f4d915a66b14/hugot_test.go#L43

To create the pure Go session that this test example uses, look here https://github.com/knights-analytics/hugot/blob/68dda1db9606c7f9cc9f93c07387f4d915a66b14/hugot_go_test.go#L11

[devlux76]

Hi @devlux76, as @janpfeifer says, I think hugot might be great for what you are asking, we essentially take the goMLX backend and wrap it with tokenizers and huggingface pipeline logic so that you can go straight from e.g. strings to model output. Take a look at this example from our tests https://github.com/knights-analytics/hugot/blob/68dda1db9606c7f9cc9f93c07387f4d915a66b14/hugot_test.go#L43

To create the pure Go session that this test example uses, look here https://github.com/knights-analytics/hugot/blob/68dda1db9606c7f9cc9f93c07387f4d915a66b14/hugot_go_test.go#L11

Thanks! I originally came here from trying hugot and failing with the downloads to see if I could figure out the problem. Got the problem sorted then realized I needed to figure out tokenization which is what spurred this question. I'm still trying to determine if the problem I was having with hugot was with the downloader or the model(s) I was trying to download. There seems to be a bug in the hugot downloader that won't visit hugging face but instead looks locally for the file and bails out. However if I specify the protocol as huggingface: then it tries to get the model from hugging face but constructs a bad url that includes the word huggingface: in it. This doesn't happen in the go-huggingface downloader when I use it directly. I'm still a bit of a noob with golang, otherwise I'd put more effort in and try to file a bug report.

[RJKeevil]

Hi @devlux76, interestingly hugot uses the same downloader as goMLX, but yes it checks locally to see if the file already exists. If you send me an example and/or a log message i can probably spot whats up.