Intermittently returns 503 when routing to ExternalName service (same-cluster, cross-namespace) #14565
Description
Is there an existing issue for this?
- I have searched the existing issues
Kong version ($ kong version)
Kong 3.8
Current Behavior
In Kong 3.8 deployed on EKS, when using a Kubernetes ExternalName service (resolving to another service in a different namespace within the same cluster), Kong intermittently returns 503 Service Unavailable responses—about 50% of the time—when routing traffic via a Kubernetes Ingress or a Gateway API HTTPRoute.
If the HTTPRoute uses a BackendRef that directly targets the actual service in the other namespace, bypassing the ExternalName indirection, the issue does not occur.
Also curl the target service hostname manually from the Kong pod do not show a 503 at all.
While setting kong in debug I do not observe any specific message:
2025/05/22 16:23:31 [debug] 1406#0: *12680 [lua] init.lua:1397: balancer(): setting address (try 1): 172.20.159.184:80
2025/05/22 16:23:31 [debug] 1406#0: *12680 [lua] init.lua:1430: balancer(): enabled connection keepalive (pool=172.20.159.184|80, pool_size=512, idle_timeout=60, max_requests=10000)
10.250.11.54 - - [22/May/2025:16:23:31 +0000] "POST /api HTTP/1.1" 503 107 "-" "curl/8.7.1" kong_request_id: "f7b91609241e6716bb55a1636c5ba46f"
This is the log request I can see from the tries it identifies the correct ip of the service in the targeted namespace
{
"client_ip": "121.200.241.162",
"started_at": 1747931011686,
"route": {
"service": {
"id": "8942cb25-df0d-5569-bbb6-2f64b5d12ea2"
},
"path_handling": "v0",
"paths": [
"~/api($|/)(.*)"
],
"id": "270b826d-38cd-5536-a7cd-a317480ced93",
"preserve_host": true,
"strip_path": false,
"regex_priority": 0,
"name": "xxx.xxx-xxxxx-xxx-ingress.ooo-vendor.sei-www.myhost.com.80",
"hosts": [
"www.myhost.com"
],
"request_buffering": true,
"response_buffering": true,
"tags": [
"k8s-name:myingress-ingress",
"k8s-namespace:mnamespace",
"k8s-kind:Ingress",
"k8s-uid:6430d3b3-f5a5-40f7-8e81-879f271a8af4",
"k8s-group:networking.k8s.io",
"k8s-version:v1"
],
"https_redirect_status_code": 426,
"ws_id": "0dc6f45b-8f8d-40d2-a504-473544ee190b",
"created_at": 1747930762,
"updated_at": 1747930762,
"protocols": [
"http",
"https"
]
},
"response": {
"size": 400,
"headers": {
"server": "kong/3.8.0",
"via": "1.1 kong/3.8.0",
"content-length": "107",
"x-kong-upstream-latency": "0",
"x-kong-request-id": "f7b91609241e6716bb55a1636c5ba46f",
"content-type": "text/html; charset=UTF-8",
"connection": "close",
"cache-control": "no-cache",
"x-kong-proxy-latency": "0"
},
"status": 503
},
"kong_plugin": "my-file-log",
"workspace_name": "default",
"upstream_uri": "/",
"tries": [
{
"port": 80,
"balancer_start": 1747931011686,
"balancer_start_ns": 1747931011686800000,
"balancer_latency_ns": 14592,
"ip": "172.20.159.184",
"balancer_latency": 0
}
],
"workspace": "0dc6f45b-8f8d-40d2-a504-473544ee190b",
"source": "upstream",
"service": {
"write_timeout": 60000,
"enabled": true,
"tags": [
"k8s-name:service-vendor",
"k8s-namespace:mnamespace",
"k8s-kind:Service",
"k8s-uid:720a4962-0339-4d64-b782-486d63825550",
"k8s-version:v1"
],
"id": "8942cb25-df0d-5569-bbb6-2f64b5d12ea2",
"port": 80,
"name": "mnamespace.service-vendor.80",
"connect_timeout": 60000,
"read_timeout": 60000,
"host": "service-vendor.80.svc",
"created_at": 1747930762,
"updated_at": 1747930762,
"retries": 5,
"path": "/",
"protocol": "http",
"ws_id": "0dc6f45b-8f8d-40d2-a504-473544ee190b"
},
"upstream_status": "503",
"latencies": {
"kong": 0,
"proxy": 0,
"receive": 1,
"request": 1
},
"request": {
"uri": "/api",
"querystring": {},
"tls": {
"cipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
"client_verify": "NONE",
"version": "TLSv1.2"
},
"url": "http://www.myhost.com:80/api",
"size": 392,
"headers": {
"user-agent": "curl/8.7.1",
"accept": "*/*",
"content-length": "91",
"host": "www.myhost.com",
"content-type": "application/json"
},
"method": "POST",
"id": "f7b91609241e6716bb55a1636c5ba46f"
}
}
Expected Behavior
Kong should route all requests to the target service without intermittent failure.
Steps To Reproduce
service.yaml config
apiVersion: v1
kind: Service
metadata:
name: service-vendor
spec:
externalName: m_service.m_namespace.svc.cluster.local.
ports:
- port: 80
protocol: TCP
targetPort: 80
sessionAffinity: None
type: ExternalName
ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
konghq.com/plugins: transformer-path
konghq.com/regex-prefix: /~
name: myingress
spec:
ingressClassName: kong
rules:
- host: www.myhost.com
http:
paths:
- backend:
service:
name: service-vendor
port:
number: 8545
path: /~/api($|/)(.*)
pathType: ImplementationSpecific
Anything else?
No response