"docker model run" optimization

[ericcurtin]

Some inference engines are faster to start than others. A typical example of one that is slow to start in vLLM. This is ok for data centres. They are happy to start a service, warm it up properly, etc. and leave it running basically forever. Being optimal after startup is the focus. When we do:

docker model run some_model

today. We only start vllm server once we have completed typing our first query and press enter to send it. This is slow.

What we should do instead is when the user does:

docker model run some_model

We should start the "vllm serve" process immediately in the background, so by the time the user sends the first query, they will have to wait less, it may even be fully loaded at that point. RamaLama does this FWIW (I implemented this there)

[ericcurtin]

It's probably a different issue, but there should probably be a docker-compose equivalent of this. On docker compose up, load this model immediately (or load these two models immediately). Don't wait for the first request.

"on demand" vs "do it now" (can support both)

"do it now" is sometimes better for production.

[doringeman]

I can take this.

[ericcurtin]

I can take this.

ollama also has a similar concept:

ollama run model ""

Just load a model, don't do inference.

But I think it waits for the model to load. We don't want to wait.

[doringeman]

@ericcurtin see #629. WDYT? It works for Compose, too.