Raven IPsec L3 tunnel: traffic from non-gateway nodes to edge Pod CIDRs fails when forwarded via gateway

[Sabre94]

In an OpenYurt cluster using Raven for cloud-edge connectivity (IPsec L3 tunnel), only the cloud gateway node can reach the edge Pod network. Traffic from other cloud nodes to edge Pod IPs times out, and edge-to-non-gateway-cloud nodes also fails.

Example symptom: Prometheus running on a non-gateway cloud node cannot scrape a metrics endpoint on an edge node:

Error scraping target: Get "http://Pod_IP:9400/metrics": context deadline exceeded

Observed connectivity:

• cloud gateway node (master-gw) <-> edge: OK
• other cloud nodes (master-other) -> edge Pod IPs: FAIL
• edge -> master-other Pod IPs: FAIL

Flannel MASQUERADE SNAT breaks xfrm policy matching (tunnel not triggered)

On the cloud gateway node , Flannel installs a MASQUERADE rule similar to:

-A FLANNEL-POSTRTG -s 10.16.0.0/12 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully

When traffic from a non-gateway node Pod (e.g. 10.16.2.x) is forwarded via master-gw towards the edge Pod CIDR (10.16.10.0/24), it hits nat/POSTROUTING on master-gw and is SNATed to master-gw’s node IP. After SNAT, the original (src=10.16.2.0/24, dst=10.16.10.0/24) no longer matches the xfrm policy, so the IPsec tunnel is not triggered.

We saw:

• ip xfrm policy exists for src 10.16.2.0/24 -> dst 10.16.10.0/24 (dir out), but lifetime current stays 0
• ip -s xfrm state shows OUT SA counters for that subnet pair stay at 0

Workaround

On the cloud gateway node (master-gw), inserting a “podCIDR -> podCIDR skip SNAT” rule at the top of nat/POSTROUTING fixes the SNAT/xfrm mismatch:

iptables -t nat -I POSTROUTING 1 -s 10.16.0.0/12 -d 10.16.0.0/12 -j RETURN

After this, traffic from master-02/03 to edge Pod IPs works and xfrm counters start increasing.

We also observed that disabling VPC “source/destination check” on the gateway NIC can restore connectivity (even without the above iptables rule), implying VPC forwarding restrictions can be a second factor.

[Sabre94]

Plan A: Current branch

Goal: avoid unwanted SNAT for cross‑gateway traffic and eliminate brief disconnects caused by periodic rule rebuilds
Flow:

1. Collect subnets from the local gateway and all remote gateways (Subnets).
2. Build a supernet: compute the minimal supernet that covers all subnets, then downscale to /12 (if the computed supernet is already larger than /12, keep it and log a warning).
3. Fix the CIDR: once the /12 supernet is determined, keep it in memory for the process lifetime. Subsequent reconciliations do not recalculate or clear chains.
4. Install rules: insert a jump rule at the top of POSTROUTING to RAVEN-CROSSGW-SKIPNAT, and add -s <CIDR> -d <CIDR> -j ACCEPT in that chain to bypass SNAT.

Plan B:

Goal: fully eliminate runtime recomputation and rely on SRE‑defined CIDR planning.
Flow:

1. Add a deployment parameter (e.g., --cross-gateway-skip-nat-cidr) to specify a fixed CIDR like /12.
2. Use the fixed CIDR directly to generate skip‑SNAT rules, without any runtime merge/compute.
3. Operational constraint: ensure all cluster/gateway subnets stay inside the declared CIDR.

[zyjhtangtang]

@Sabre94 Great! SNAT rules indeed affect IPsec connectivity. For L3 gateway nodes, collect all subnets from remote gateways and merge them (not necessarily into a single supernet—multiple subnets are acceptable). Then, configure SNAT bypass rules for each of the merged subnets.