feat(ai-proxy): add path_mode option to override.endpoint #12857
feat(ai-proxy): add path_mode option to override.endpoint #12857sihyeonn wants to merge 4 commits intoapache:masterfrom
Conversation
Add `override.path_mode` option to control upstream request path: - fixed (default): use endpoint path or driver default - preserve: use original request URI path with query string - append: append original request URI path to endpoint path This allows using a single route for multiple API endpoints (e.g., /v1/chat/completions and /v1/responses) with the same model, reducing route explosion in multi-model deployments. Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>
Add test cases for ai-proxy and ai-proxy-multi: - path_mode=preserve: preserves original request URI path - path_mode=append: appends request path to endpoint path - path_mode=fixed: uses endpoint path (existing behavior) - query string preservation with preserve/append modes Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>
dosubot
bot
added
size:L
sihyeonn
changed the title
|
Hi @sihyeonn, thank you for your contribution. Could you please describe your original requirements in detail? What problems did you encounter while using the current version, and why was this change necessary? |
|
Hi @Baoyuantop , I'm running an LLM gateway. Many models expose multiple endpoints from the same base URL:
Currently, I must create separate routes for each endpoint because # Example Current: 4 routes per model
routes:
- uri: /v1/chat/completions
plugins:
ai-proxy:
override:
endpoint: https://api.example.com/models/gpt-4/v1/chat/completions
- uri: /v1/responses
plugins:
ai-proxy:
override:
endpoint: https://api.example.com/models/gpt-4/v1/responses
# ... 2 more routesThis creates unnecessary duplication:
|
| if ctx.var.is_args and ctx.var.args and #ctx.var.args > 0 then | ||
| local req_args_tab = core.string.decode_args(ctx.var.args) | ||
| if type(req_args_tab) == "table" then | ||
| core.table.merge(query_params, req_args_tab) | ||
| end |
There was a problem hiding this comment.
It can be extracted into a common function.
There was a problem hiding this comment.
It is recommended to use ctx.var.is_args == "?" to check.
| end | ||
| end | ||
| else | ||
| path = (endpoint_path and endpoint_path ~= "" and endpoint_path) or self.path |
There was a problem hiding this comment.
It needs to be confirmed whether the behavior remains consistent when endpoint_path is an empty string.
apisix/plugins/ai-proxy/schema.lua
Outdated
| type = "string", | ||
| description = "To be specified to override the endpoint of the AI Instance", | ||
| }, | ||
| path_mode = { |
There was a problem hiding this comment.
It is recommended to extract it as a common schema variable to avoid duplication.
Baoyuantop
added
the
wait for update
- Extract merge_request_query_params() to avoid code duplication - Use ctx.var.is_args == "?" for proper string comparison - Extract path_mode_schema as shared variable to avoid duplication - Clarify empty string handling in fixed mode path selection Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>
|
@Baoyuantop Thanks for your review. I handled it. |
|
Hi @sihyeonn, please help resolve the code conflicts. |
|
|
||
|
|
||
| function _M.new(opts) | ||
| local self = { |
There was a problem hiding this comment.
Is this change necessary? It breaks the original transparent pass-through mechanism—now only explicitly listed fields are passed through.
2 participants
Description
Add
override.path_modeoption toai-proxyandai-proxy-multiplugins to control how the upstream request path is determined.Background (Problem)
Currently, when
override.endpointis set, the URL path is fixed to either the endpoint's path or the driver's default path. This causes issues when:/v1/chat/completionsand/v1/responses)Solution
Add
path_modeoption with three modes:fixed(default): Existing behavior - uses endpoint path or driver defaultpreserve: Uses the original request URI path (with query string)append: Appends original request URI path to the endpoint pathThis allows using a single route with the same model for multiple API endpoints, reducing route explosion in multi-model deployments.
Which issue(s) this PR fixes:
Fixes #
Checklist