[BUG] Can't use embedding models from ollama

[zhangphi]

Describe the bug

I launched apeRAG and all databases from docker containers,then I tried to use embedding models from Ollama, but failed.

To Reproduce

Steps to reproduce the behavior:

1. Start local Ollama service.
2. Models->Model Providers->Add Provider, add 'ollama' with 'Base URL'='http://host.docker.internal:11434/v1'
3. Set up an embedding model for ollama: nomic-embed-text:latest
4. Create a collection 'TechCorp2' and set up the Embedding model with nomic-embed-text:latest and Completion model gpt-4o-mini.
5. Upload a simple text file to test: Both 'Vector Store' and 'Graph extraction' falied.

Screenshots & Logs

In a terminal,
$ sudo docker logs aperag-celeryworker
gave me the following errors:
......
[2025-09-11 01:58:54,903: ERROR/MainProcess] aperag.llm.embed.embedding_service - Batch embedding API call failed: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers
[2025-09-11 01:58:54,903: ERROR/MainProcess] aperag.llm.embed.embedding_service - Batch embedding with indices failed: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'})
[2025-09-11 01:58:54,903: ERROR/MainProcess] aperag.llm.embed.embedding_service - Batch processing failed: Embedding API error: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'}) (Details: {'original_error': "Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'})"})
[2025-09-11 01:58:54,903: ERROR/MainProcess] aperag.llm.embed.base_embedding - Failed to probe embedding dimension for provider_db179b07/nomic-embed-text:latest: Batch processing error (batch size: 10): Failed to process 1 batches: ['Embedding API error: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'}) (Details: {'original_error': "Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'})"})'] contents: ['dimension_probe'] (Details: {'batch_size': 10, 'reason': 'Failed to process 1 batches: ['Embedding API error: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {\'original_error\': \'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers\'}) (Details: {\'original_error\': "Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {\'original_error\': \'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers\'})"})'] contents: ['dimension_probe']'})
[2025-09-11 01:58:54,903: ERROR/MainProcess] aperag.graph.lightrag_manager - Failed to create embedding function - configuration error: Failed to probe embedding dimension: Batch processing error (batch size: 10): Failed to process 1 batches: ['Embedding API error: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'}) (Details: {'original_error': "Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {'original_error': 'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers'})"})'] contents: ['dimension_probe'] (Details: {'batch_size': 10, 'reason': 'Failed to process 1 batches: ['Embedding API error: Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {\'original_error\': \'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers\'}) (Details: {\'original_error\': "Embedding API error: litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers (Details: {\'original_error\': \'litellm.BadRequestError: Unmapped LLM provider for this endpoint. You passed model=provider_db179b07/nomic-embed-text:latest, custom_llm_provider=provider_db179b07. Check supported provider and route: https://docs.litellm.ai/docs/providers\'})"})'] contents: ['dimension_probe']'}) (Details: {'provider': 'provider_db179b07', 'model': 'nomic-embed-text:latest'})
.......

Additional context

• OS: Ubuntu 22.04
• Browser: Google Chrome
• Version: 22.04
• ApeRAG docker version: apecloud/aperag v0.0.0-nightly 939cc6150243 41 hours ago

[zhangphi]

$ sudo docker exec -it aperag-api curl -v http://host.docker.internal:11434/api/tags

Direct testing the embedding with Ollama works fine:
$ curl http://localhost:11434/api/embeddings
-H "Content-Type: application/json"
-d '{
"model": "nomic-embed-text:latest",
"prompt": "Llamas are members of the camelid family"
}'

[zhangphi]

Another thing I found quite strange is:
After I update the embedding model and completion model on the collection settings, it looks OK:

but then if I hard/hand fresh the browser, the embedding model dissapears:

But if I check the celerayworker log, the system seems able to find the right embedding model, but gets stuck with something else.

[zhangphi]

It would be great if someone provides documentation or guideline on how to set up local LLMs/embedding models for ApeRAG! Huge thanks!

[zhangphi]

In a terminal, I tested the text embedding within docker container aperag-api with the local ollama service, and it works fine:

[sailwebs]

Another thing I found quite strange is: After I update the embedding model and completion model on the collection settings, it looks OK:

but then if I hard/hand fresh the browser, the embedding model dissapears: But if I check the celeryworker log, the system seems able to find the right embedding model, but gets stuck with something else.

The issue where the model was not displayed has been fixed in this PR:
ab30956

[zhangphi]

Another thing I found quite strange is: After I update the embedding model and completion model on the collection settings, it looks OK:

but then if I hard/hand fresh the browser, the embedding model dissapears:

But if I check the celeryworker log, the system seems able to find the right embedding model, but gets stuck with something else.

The issue where the model was not displayed has been fixed in this PR: ab30956

Thanks a lot! @sailwebs

[zhangphi]

With the help of deepwiki and gemini-cli, I have figured out how to fix that.

The key missing part is: when I add a model provider that is not in the predefined providers, ApeRAG will assign a random name to the new provider, like provider_db179b07 in my case, and this will cause some troubles for LiteLLM, because LiteLLM doesn't know how to handle it.

Fix: Open aperag/service/llm_provider_service.py, in the definition of function create_llm_provider(),

async def create_llm_provider(provider_data: dict, user_id: str, is_admin: bool = False):
    """Create a new LLM provider or restore a soft-deleted one with the same name

    Args:
        provider_data: Provider configuration data
        user_id: User ID creating the provider
        is_admin: Whether the user is an admin
    """
    # add the following code:
    # If name is not provided and base_url contains port "11434", set dialects to "ollama"
    if "name" not in provider_data or not provider_data["name"]:
        base_url = provider_data.get("base_url", "")
        try:
            parsed_url = urlparse(base_url)
            # Check for Ollama's default port (11434)
            # if ":11434" in base_url or "localhost:11434" in base_url or "127.0.0.1:11434" in base_url or "http://host.docker.internal:11434" in base_url:
            if parsed_url.port == 11434:
                provider_data["name"] = "ollama"
                provider_data["completion_dialect"] = "ollama"
                provider_data["embedding_dialect"] = "ollama"
        except:
            pass  # Handle invalid URLs gracefully
     ......

Then launch the services from source code, and add Ollama as a model provider with Base Url=http://localhost:11434 in Settings->Models:

Now ApeRAG works fine with (embedding and/or completion) models from Ollama.

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity