Feat/support multi providers ， OpenViking支持多providers (#192)

* support multi-provider

* revise readme, fix bugs

Author: BytedanceFu

.gitignore

@@ -168,3 +168,4 @@ benchmark_stress_db/
 examples/data/
 third_party/agfs/bin/
 openviking/_version.py
+specs/

README.md

@@ -89,10 +89,115 @@ OpenViking requires the following model capabilities:
 - **VLM Model**: For image and content understanding
 - **Embedding Model**: For vectorization and semantic retrieval
 
-OpenViking supports various model services:
-- **OpenAI Models**: Supports GPT-4V and other VLM models, and OpenAI Embedding models.
-- **Volcengine (Doubao Models)**: Recommended for low cost and high performance, with free quotas for new users. For purchase and activation, please refer to: [Volcengine Purchase Guide](./docs/en/guides/02-volcengine-purchase-guide.md).
-- **Other Custom Model Services**: Supports model services compatible with the OpenAI API format.
+#### Supported VLM Providers
+
+OpenViking supports multiple VLM providers:
+
+| Provider | Model | Get API Key |
+|----------|-------|-------------|
+| `volcengine` | doubao | [Volcengine Console](https://console.volcengine.com/ark) |
+| `openai` | gpt | [OpenAI Platform](https://platform.openai.com) |
+| `anthropic` | claude | [Anthropic Console](https://console.anthropic.com) |
+| `deepseek` | deepseek | [DeepSeek Platform](https://platform.deepseek.com) |
+| `gemini` | gemini | [Google AI Studio](https://aistudio.google.com) |
+| `moonshot` | kimi | [Moonshot Platform](https://platform.moonshot.cn) |
+| `zhipu` | glm | [Zhipu Open Platform](https://open.bigmodel.cn) |
+| `dashscope` | qwen | [DashScope Console](https://dashscope.console.aliyun.com) |
+| `minimax` | minimax | [MiniMax Platform](https://platform.minimax.io) |
+| `openrouter` | (any model) | [OpenRouter](https://openrouter.ai) |
+| `vllm` | (local model) | — |
+
+> 💡 **Tip**: OpenViking uses a **Provider Registry** for unified model access. The system automatically detects the provider based on model name keywords, so you can switch between providers seamlessly.
+
+#### Provider-Specific Notes
+
+<details>
+<summary><b>Volcengine (Doubao)</b></summary>
+
+Volcengine supports both model names and endpoint IDs. Using model names is recommended for simplicity:
+
+```json
+{
+  "vlm": {
+    "provider": "volcengine",
+    "model": "doubao-seed-1-6-240615",
+    "api_key": "your-api-key"
+  }
+}
+```
+
+You can also use endpoint IDs (found in [Volcengine ARK Console](https://console.volcengine.com/ark)):
+
+```json
+{
+  "vlm": {
+    "provider": "volcengine",
+    "model": "ep-20241220174930-xxxxx",
+    "api_key": "your-api-key"
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Zhipu AI (智谱)</b></summary>
+
+If you're on Zhipu's coding plan, use the coding API endpoint:
+
+```json
+{
+  "vlm": {
+    "provider": "zhipu",
+    "model": "glm-4-plus",
+    "api_key": "your-api-key",
+    "api_base": "https://open.bigmodel.cn/api/coding/paas/v4"
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>MiniMax (中国大陆)</b></summary>
+
+For MiniMax's mainland China platform (minimaxi.com), specify the API base:
+
+```json
+{
+  "vlm": {
+    "provider": "minimax",
+    "model": "abab6.5s-chat",
+    "api_key": "your-api-key",
+    "api_base": "https://api.minimaxi.com/v1"
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Local Models (vLLM)</b></summary>
+
+Run OpenViking with your own local models using vLLM:
+
+```bash
+# Start vLLM server
+vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
+```
+
+```json
+{
+  "vlm": {
+    "provider": "vllm",
+    "model": "meta-llama/Llama-3.1-8B-Instruct",
+    "api_key": "dummy",
+    "api_base": "http://localhost:8000/v1"
+  }
+}
+```
+
+</details>
 
 ### 3. Environment Configuration
 
@@ -106,20 +211,22 @@ Create a configuration file `~/.openviking/ov.conf`:
     "dense": {
       "api_base" : "<api-endpoint>",   // API endpoint address
       "api_key"  : "<your-api-key>",   // Model service API Key
-      "provider" : "<provider-type>",  // Provider type (volcengine or openai)
+      "provider" : "<provider-type>",  // Provider type: "volcengine" or "openai" (currently supported)
       "dimension": 1024,               // Vector dimension
       "model"    : "<model-name>"      // Embedding model name (e.g., doubao-embedding-vision-250615 or text-embedding-3-large)
     }
   },
   "vlm": {
     "api_base" : "<api-endpoint>",     // API endpoint address
     "api_key"  : "<your-api-key>",     // Model service API Key
-    "provider" : "<provider-type>",    // Provider type (volcengine or openai)
+    "provider" : "<provider-type>",    // Provider type (volcengine, openai, deepseek, anthropic, etc.)
     "model"    : "<model-name>"        // VLM model name (e.g., doubao-seed-1-8-251228 or gpt-4-vision-preview)
   }
 }
 ```
 
+> **Note**: For embedding models, currently only `volcengine` (Doubao) and `openai` providers are supported. For VLM models, we support multiple providers including volcengine, openai, deepseek, anthropic, gemini, moonshot, zhipu, dashscope, minimax, and more.
+
 #### Configuration Examples
 
 👇 Expand to see the configuration example for your model service:

openviking/models/vlm/__init__.py

@@ -2,13 +2,29 @@
 # SPDX-License-Identifier: Apache-2.0
 """VLM (Vision-Language Model) module"""
 
+from .backends.litellm_vlm import LiteLLMVLMProvider
 from .backends.openai_vlm import OpenAIVLM
 from .backends.volcengine_vlm import VolcEngineVLM
 from .base import VLMBase, VLMFactory
+from .registry import (
+    PROVIDERS,
+    ProviderSpec,
+    find_by_model,
+    find_by_name,
+    find_gateway,
+    get_all_provider_names,
+)
 
 __all__ = [
     "VLMBase",
     "VLMFactory",
     "OpenAIVLM",
     "VolcEngineVLM",
+    "LiteLLMVLMProvider",
+    "ProviderSpec",
+    "PROVIDERS",
+    "find_by_model",
+    "find_by_name",
+    "find_gateway",
+    "get_all_provider_names",
 ]