london switches have lost their mind.

[leftwo]

After an mupdate to a recent omicron commit, both switches in London are having trouble communicating with the outside world. Everything came back after mupdate, but NTP failed to sync on the (already setup) racklette.

A mupdate of berlin, also using a bgp config, and it's having trouble as well. Berlin received omicron commit 9ccd2038cf5bff650ba0772b55d16bc75b6b0e15. Berlin was stuck during the RSS NTP sync phase.

[leftwo]

London was using omicron commit 6cef874 (plus some side stuff, unrelated I promise)

[JustinAzoff]

Also seeing similar behavior on madrid with 4a517f2

[rcgoodfellow]

On London/sw0 BGP was not establishing. We were seeing the TCP SYN packets being sent by centrum to sw0 being corrupted between when they left eveningstar (packets captured on an egress port monitor) and when they arrived in the switch zone. The nature of the corruption was a change in the TCP source port of the SYN.

The packet as observed when leaving eveningstar

0000   a8 40 25 05 02 23 aa 00 04 00 ca fe 08 00 45 c0
0010   00 3c da 57 40 00 01 06 28 44 ac 14 0f 1b ac 14
0020   0f 1d ac 31 00 b3 84 f4 27 61 00 00 00 00 a0 02
             ^^^^^
0030   83 2c a1 20 00 00 02 04 05 b4 04 02 08 0a b9 a7
0040   9a 76 00 00 00 00 01 03 03 01

and then the packet observed arriving in the switch zone.

0000   a8 40 25 05 02 23 aa 00 04 00 ca fe 08 00 45 c0
0010   00 3c da 57 40 00 01 06 28 44 ac 14 0f 1b ac 14
0020   0f 1d a0 01 00 b3 84 f4 27 61 00 00 00 00 a0 02
             ^^^^^
0030   83 2c a1 20 00 00 02 04 05 b4 04 02 08 0a b9 a7
0040   9a 76 00 00 00 00 01 03 03 01

The possible places this corruption could be happening are

1. The Tofino ASIC.
2. The Sled kernel before snoop has a chance to observe the packet.

This condition persisted for around 12 hours. Then all of a sudden things started working. I was off building a repro for the Tofino simulator. I came back to London to find BGP sessions up and a seemingly unrelated NAT forwarding issue on a separate switch had also sorted itself out. We can see the time when things started to come back to life from the Maghemite BGP peering log.

root@oxz_switch0:~# mgadm bgp hist fsm

FSM Event History - Peer: 172.20.15.27 - Major Events
====================================================================================================
Showing 8 of 8 events

Timestamp                Category         Event                        State         Transition                  Details
1986-12-28 12:35:33.358  StateTransition  SessionSetup -> Established  Established   SessionSetup → Established  State transition
1986-12-28 12:35:33.357  StateTransition  OpenConfirm -> SessionSetup  SessionSetup  OpenConfirm → SessionSetup  State transition
1986-12-28 12:35:33.355  StateTransition  OpenSent -> OpenConfirm      OpenConfirm   OpenSent → OpenConfirm      State transition
1986-12-28 12:35:33.354  StateTransition  Connect -> OpenSent          OpenSent      Connect → OpenSent          State transition
1986-12-28 12:35:33.351  Session          tcp connection acked         Connect       Connect                     inbound from 172.20.15.27:53249
1986-12-28 00:02:19.569  Admin            announce                     Connect       Connect                     -
1986-12-28 00:02:19.569  StateTransition  Idle -> Connect              Connect       Idle → Connect              State transition
1986-12-28 00:02:19.569  Admin            manual start                 Idle          Idle                        -

The dpd log file around this time appears to be

/pool/ext/036b998d-8238-4478-84a4-21a87e26e958/crypt/debug/oxz_switch/oxide-dendrite:default.log.536112901

on BRM42220036, but there is nothing remarkable at the time that BGP sprung to life.

[Nieuwejaar]

The log file covering the time with the BGP and NTP successes is /pool/ext/97bfc396-2da3-402f-b57a-9ad4b1397ccf/crypt/debug/oxz_switch/oxide-dendrite:default.log.1769151101.

I assume this is the BGP success here:

12:35:33.380Z INFO dpd: adding route 0.0.0.0/0 -> 172.20.15.27
12:35:33.380Z DEBG dpd: initted freemap.  size: 8191
    unit = freemap_route_ipv4
12:35:33.380Z DEBG dpd: reset freemap
    unit = freemap_route_ipv4
12:35:33.380Z DEBG dpd: replacing targets for 0.0.0.0/0 with: [NextHop { asic_port_id: 320, route: Route { tag: "mg-lower", port_id: Qsfp(QsfpPort(0)), link_id: LinkId(0), tgt_ip: 172.20.15.27, vlan_id: None } }]
12:35:33.380Z INFO dpd: Entering pipe_mgr_exm_

with NTP succeeding a bit later:

12:42:53.408Z ERRO dpd: BF SDE dropped receiver before we could send a response, likely due to the receive timeout
    data = Ok(Read([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))
    request = Read(ReadRequest { module: 35, bank: 0, page: 0, offset: 9, len: 13 })
06:49:34.088Z ERRO dpd: failed to send message to dpd message = Read(ReadRequest { module: 36, bank: 0, page: 0, offset: 6, len: 3 })
    reason = RecvTimeout
    unit = qsfp-ffi

There are no table updates for at least 20 minutes before BGP started working.

[Nieuwejaar]

On switch1, the NTP sync happens in /pool/ext/6e5fd695-54b2-42cf-87f1-4ff9048956a4/crypt/debug/oxz_switch/oxide-dendrite:default.log.176915201

12:45:54.176Z INFO dpd: request completed
    latency_us = 105
    local_addr = [::1]:12224
    method = GET
    remote_addr = [::1]:58839
    req_id = a29b47f1-4bbc-448b-be7a-493a2345acfa
    response_code = 200
    server_id = 2
    unit = api-server
    uri = /links/tfport_data
06:52:43.035Z DEBG dpd: determined request API version
    local_addr = [::1]:12224
    method = GET
    remote_addr = [::1]:43588
    req_id = fdd65feb-5122-46de-8dae-ec5a76b88483
    server_id = 2
    unit = api-server
    uri = /links

That is near the top of the file, and there is nothing interesting prior to that. The previous log file is /pool/ext/6e5fd695-54b2-42cf-87f1-4ff9048956a4/crypt/debug/oxz_switch/oxide-dendrite.536157900.

Just before the NTP sync we see:

12:44:56.569Z DEBG dpd: add nat entry (172.20.34.5/0-65535) -> NatTarget { internal_ip: fd14:df81:5b08:101::1, inner_mac: a8:40:25:ff:d8:8d, vni: Vni(100) }
12:44:56.569Z INFO dpd: Ingress.nat_ingress.ingress_ipv4 Request to install TCAM entry tbl 0x1000012 dev 0 pipe ffff match spec
    module = Pipe
    unit = bf-sde
12:44:56.569Z DEBG dpd: Action Spec:
    module = Pipe
    unit = bf-sde
12:44:56.569Z INFO dpd: TCAM Ingress.nat_ingress.ingress_ipv4(0x1000012 - 0) pipe 65535 ptn 0 Entry 0x5 with group     0 priority     0 allocated at     3 ( 3- 1020)
    module = Pipe
    unit = bf-sde
12:44:56.569Z DEBG dpd: Expanded match spec for block 0 entry 0
    module = Pipe
    unit = bf-sde

That certainly seems interesting.

[Nieuwejaar]

That nat entry is for the nexus zone on the other scrimlet, so it's hard to see how it would somehow get the ntp zone on this scrimlet working. Both of these events may just be signs of the system grinding to life after the original problem cleared up.

[leftwo]

I took berlin with a bgp config and tried a few different git revs to see what made it through RSS:
Worked: 4175bf6 Audit log all endpoints + coverage test (#9467)
Worked: ed44d4b [nexus] Having children shouldn't prevent reincarnation
Failed: 915ab1e bump dendrite to 181

On the failing RSS, berlin was able to get BGP routes for both switches:

root@oxz_switch0:~# mgadm bgp status neighbors 65107                          
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive                                                                                    
172.20.15.67  Some(64601)  Established  59m 15s 71ms    6s/6s  2s/2s     

root@oxz_switch1:~# mgadm bgp status neighbors 65107
Peer Address  Peer ASN     State        State Duration  Hold   Keepalive
172.20.15.67  Some(64601)  Established  59m 23s 755ms   6s/6s  2s/2s

Switch 0 shows NAT ingress, while switch 1 does not:

root@oxz_switch0:~# swadm table counters pipe.Ingress.nat_ingress.ingress_ipv4                                                                              
dst_addr     ports        Packets  Bytes                                      
172.20.30.5  0-16383      252      34676                                      
172.20.30.5  16384-32767  253      34776

root@oxz_switch1:~# swadm table counters pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports        Packets  Bytes
172.20.30.5  0-16383      0        0
172.20.30.5  16384-32767  0        0

Boundary opte ports are not getting external addresses:

root@oxz_switch0:~# pilot host exec -c '/opt/oxide/opte/bin/opteadm list-ports' 14-17                                                                       
14  BRM42220011        ok: LINK   MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE         
opte0  A8:40:25:FF:D0:F3  172.30.3.6    None            None           None          None           None           running                                                  
16  BRM42220082        ok: LINK   MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE         
opte0  A8:40:25:FF:C5:31  172.30.3.5    None            None           None          None           None           running                                  

[rcgoodfellow]

Documenting table state for unwell berlin switch0

root@oxz_switch0:~# swadm table list | xargs -t -l swadm table dump
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.lookup
dst_addr          action  args
0.0.0.0/0  index  idx=0   slots=1
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.route
idx           action                args
0    forward  nexthop=172.20.15.67  port=320
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.lookup
dst_addr                        action  args
fdb0:a840:2504:b54::/64  index  idx=0   slots=1
fdb0:a840:2504:492::/64  index  idx=1   slots=1
fdb0:a840:2504:211::/64  index  idx=2   slots=1
fdb0:a840:2504:190::/64  index  idx=3   slots=1
fde4:f501:6a7d:101::/64  index  idx=4   slots=1
fde4:f501:6a7d:103::/64  index  idx=5   slots=1
fde4:f501:6a7d:104::/64  index  idx=6   slots=1
fde4:f501:6a7d:102::/64  index  idx=7   slots=1
fde4:f501:6a7d:1::/64    index  idx=8   slots=1
fde4:f501:6a7d:2::/64    index  idx=9   slots=1
fde4:f501:6a7d:3::/64    index  idx=10  slots=1
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.route
idx           action                            args
0    forward  nexthop=fe80::aa40:25ff:fe04:b54  port=40
1    forward  nexthop=fe80::aa40:25ff:fe04:492  port=8
2    forward  nexthop=fe80::aa40:25ff:fe04:211  port=448
3    forward  nexthop=fe80::aa40:25ff:fe04:190  port=416
4    forward  nexthop=fe80::aa40:25ff:fe04:190  port=416
5    forward  nexthop=fe80::aa40:25ff:fe04:492  port=8
6    forward  nexthop=fe80::aa40:25ff:fe04:b54  port=40
7    forward  nexthop=fe80::aa40:25ff:fe04:211  port=448
8    forward  nexthop=fe80::aa40:25ff:fe04:190  port=416
9    forward  nexthop=fe80::aa40:25ff:fe04:211  port=448
10   forward  nexthop=fe80::aa40:25ff:fe04:492  port=8
swadm table dump pipe.Ingress.l3_router.MulticastRouter4.tbl
swadm table dump pipe.Ingress.l3_router.MulticastRouter6.tbl
swadm table dump pipe.Ingress.l3_router.Arp.tbl
ip                     action  args
172.20.15.69  rewrite  dst_mac=a8:40:25:05:35:23
172.20.15.67  rewrite  dst_mac=aa:00:04:00:ca:fe
swadm table dump pipe.Ingress.l3_router.Ndp.tbl
ip                                  action  args
fe80::aa40:25ff:fe05:3502  rewrite  dst_mac=a8:40:25:05:35:02
fe80::aa40:25ff:fe05:3503  rewrite  dst_mac=a8:40:25:05:35:03
fe80::aa40:25ff:fe05:3504  rewrite  dst_mac=a8:40:25:05:35:04
fe80::aa40:25ff:fe05:350d  rewrite  dst_mac=a8:40:25:05:35:0d
fe80::aa40:25ff:fe05:350e  rewrite  dst_mac=a8:40:25:05:35:0e
fe80::aa40:25ff:fe05:350f  rewrite  dst_mac=a8:40:25:05:35:0f
fe80::aa40:25ff:fe05:3510  rewrite  dst_mac=a8:40:25:05:35:10
fe80::aa40:25ff:fe05:3511  rewrite  dst_mac=a8:40:25:05:35:11
fe80::aa40:25ff:fe05:3512  rewrite  dst_mac=a8:40:25:05:35:12
fe80::aa40:25ff:fe05:3513  rewrite  dst_mac=a8:40:25:05:35:13
fe80::aa40:25ff:fe05:3514  rewrite  dst_mac=a8:40:25:05:35:14
fe80::aa40:25ff:fe05:3515  rewrite  dst_mac=a8:40:25:05:35:15
fe80::aa40:25ff:fe05:3516  rewrite  dst_mac=a8:40:25:05:35:16
fe80::aa40:25ff:fe05:3517  rewrite  dst_mac=a8:40:25:05:35:17
fe80::aa40:25ff:fe05:3505  rewrite  dst_mac=a8:40:25:05:35:05
fe80::aa40:25ff:fe05:3506  rewrite  dst_mac=a8:40:25:05:35:06
fe80::aa40:25ff:fe05:3507  rewrite  dst_mac=a8:40:25:05:35:07
fe80::aa40:25ff:fe05:3508  rewrite  dst_mac=a8:40:25:05:35:08
fe80::aa40:25ff:fe05:3509  rewrite  dst_mac=a8:40:25:05:35:09
fe80::aa40:25ff:fe05:350a  rewrite  dst_mac=a8:40:25:05:35:0a
fe80::aa40:25ff:fe05:350b  rewrite  dst_mac=a8:40:25:05:35:0b
fe80::aa40:25ff:fe05:3518  rewrite  dst_mac=a8:40:25:05:35:18
fe80::aa40:25ff:fe05:3519  rewrite  dst_mac=a8:40:25:05:35:19
fe80::aa40:25ff:fe05:351a  rewrite  dst_mac=a8:40:25:05:35:1a
fe80::aa40:25ff:fe05:351b  rewrite  dst_mac=a8:40:25:05:35:1b
fe80::aa40:25ff:fe05:351c  rewrite  dst_mac=a8:40:25:05:35:1c
fe80::aa40:25ff:fe05:351d  rewrite  dst_mac=a8:40:25:05:35:1d
fe80::aa40:25ff:fe05:351e  rewrite  dst_mac=a8:40:25:05:35:1e
fe80::aa40:25ff:fe05:351f  rewrite  dst_mac=a8:40:25:05:35:1f
fe80::aa40:25ff:fe05:3520  rewrite  dst_mac=a8:40:25:05:35:20
fe80::aa40:25ff:fe05:3521  rewrite  dst_mac=a8:40:25:05:35:21
fe80::aa40:25ff:fe05:3522  rewrite  dst_mac=a8:40:25:05:35:22
fe80::aa40:25ff:fe05:350c  rewrite  dst_mac=a8:40:25:05:35:0c
fe80::aa40:25ff:fe04:211   rewrite  dst_mac=a8:40:25:04:02:11
fe80::aa40:25ff:fe04:b54   rewrite  dst_mac=a8:40:25:04:0b:54
fe80::aa40:25ff:fe04:492   rewrite  dst_mac=a8:40:25:04:04:92
fe80::aa40:25ff:fe04:190   rewrite  dst_mac=a8:40:25:04:01:90
swadm table dump pipe.Ingress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:35:02
428   rewrite  mac=a8:40:25:05:35:03
36    rewrite  mac=a8:40:25:05:35:04
424   rewrite  mac=a8:40:25:05:35:05
32    rewrite  mac=a8:40:25:05:35:06
444   rewrite  mac=a8:40:25:05:35:07
20    rewrite  mac=a8:40:25:05:35:08
440   rewrite  mac=a8:40:25:05:35:09
16    rewrite  mac=a8:40:25:05:35:0a
436   rewrite  mac=a8:40:25:05:35:0b
28    rewrite  mac=a8:40:25:05:35:0c
432   rewrite  mac=a8:40:25:05:35:0d
24    rewrite  mac=a8:40:25:05:35:0e
452   rewrite  mac=a8:40:25:05:35:0f
12    rewrite  mac=a8:40:25:05:35:10
448   rewrite  mac=a8:40:25:05:35:11
8     rewrite  mac=a8:40:25:05:35:12
416   rewrite  mac=a8:40:25:05:35:13
40    rewrite  mac=a8:40:25:05:35:14
420   rewrite  mac=a8:40:25:05:35:15
44    rewrite  mac=a8:40:25:05:35:16
400   rewrite  mac=a8:40:25:05:35:17
56    rewrite  mac=a8:40:25:05:35:18
404   rewrite  mac=a8:40:25:05:35:19
60    rewrite  mac=a8:40:25:05:35:1a
408   rewrite  mac=a8:40:25:05:35:1b
48    rewrite  mac=a8:40:25:05:35:1c
412   rewrite  mac=a8:40:25:05:35:1d
52    rewrite  mac=a8:40:25:05:35:1e
392   rewrite  mac=a8:40:25:05:35:1f
64    rewrite  mac=a8:40:25:05:35:20
396   rewrite  mac=a8:40:25:05:35:21
68    rewrite  mac=a8:40:25:05:35:22
320   rewrite  mac=a8:40:25:05:35:23
swadm table dump pipe.Ingress.filter.switch_ipv4_addr
dst_addr      in_port               action  args
172.20.15.69  0x140/0x1ff  claimv4
172.20.15.69  0x0/0x0      dropv4
swadm table dump pipe.Ingress.filter.switch_ipv6_addr
dst_addr                   in_port               action  args
fdb1:a840:2504:211::1      0x2/0x1ff    claimv6
fdb1:a840:2504:211::1      0x0/0x0      dropv6
fdb2:a840:2504:211::1      0x2/0x1ff    claimv6
fdb2:a840:2504:211::1      0x0/0x0      dropv6
fdce:65b8:d370:e92::1      0x0/0x0      claimv6
fe80::aa40:25ff:fe05:3502  0x2/0x1ff    claimv6
fe80::aa40:25ff:fe05:3502  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3503  0x1ac/0x1ff  claimv6
fe80::aa40:25ff:fe05:3503  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350d  0x1b0/0x1ff  claimv6
fe80::aa40:25ff:fe05:350d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350e  0x18/0x1ff   claimv6
fe80::aa40:25ff:fe05:350e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350f  0x1c4/0x1ff  claimv6
fe80::aa40:25ff:fe05:350f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3510  0xc/0x1ff    claimv6
fe80::aa40:25ff:fe05:3510  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3511  0x1c0/0x1ff  claimv6
fe80::aa40:25ff:fe05:3511  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3512  0x8/0x1ff    claimv6
fe80::aa40:25ff:fe05:3512  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3513  0x1a0/0x1ff  claimv6
fe80::aa40:25ff:fe05:3513  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3514  0x28/0x1ff   claimv6
fe80::aa40:25ff:fe05:3514  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3515  0x1a4/0x1ff  claimv6
fe80::aa40:25ff:fe05:3515  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3516  0x2c/0x1ff   claimv6
fe80::aa40:25ff:fe05:3516  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3504  0x24/0x1ff   claimv6
fe80::aa40:25ff:fe05:3504  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3517  0x190/0x1ff  claimv6
fe80::aa40:25ff:fe05:3517  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3518  0x38/0x1ff   claimv6
fe80::aa40:25ff:fe05:3518  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3519  0x194/0x1ff  claimv6
fe80::aa40:25ff:fe05:3519  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351a  0x3c/0x1ff   claimv6
fe80::aa40:25ff:fe05:351a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351b  0x198/0x1ff  claimv6
fe80::aa40:25ff:fe05:351b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351c  0x30/0x1ff   claimv6
fe80::aa40:25ff:fe05:351c  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351d  0x19c/0x1ff  claimv6
fe80::aa40:25ff:fe05:351d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351e  0x34/0x1ff   claimv6
fe80::aa40:25ff:fe05:351e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:351f  0x188/0x1ff  claimv6
fe80::aa40:25ff:fe05:351f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3520  0x40/0x1ff   claimv6
fe80::aa40:25ff:fe05:3520  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3505  0x1a8/0x1ff  claimv6
fe80::aa40:25ff:fe05:3505  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3521  0x18c/0x1ff  claimv6
fe80::aa40:25ff:fe05:3521  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3522  0x44/0x1ff   claimv6
fe80::aa40:25ff:fe05:3522  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3506  0x20/0x1ff   claimv6
fe80::aa40:25ff:fe05:3506  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3507  0x1bc/0x1ff  claimv6
fe80::aa40:25ff:fe05:3507  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3508  0x14/0x1ff   claimv6
fe80::aa40:25ff:fe05:3508  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:3509  0x1b8/0x1ff  claimv6
fe80::aa40:25ff:fe05:3509  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350a  0x10/0x1ff   claimv6
fe80::aa40:25ff:fe05:350a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350b  0x1b4/0x1ff  claimv6
fe80::aa40:25ff:fe05:350b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:350c  0x1c/0x1ff   claimv6
fe80::aa40:25ff:fe05:350c  0x0/0x0      dropv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports                         action                       args
172.20.30.5  0-16383      forward_ipv4_to  inner_mac=a8:40:25:ff:c5:31  target=fde4:f501:6a7d:101::1  vni=100
172.20.30.5  16384-32767  forward_ipv4_to  inner_mac=a8:40:25:ff:d0:f3  target=fde4:f501:6a7d:102::1  vni=100
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6
swadm table dump pipe.Ingress.nat_ingress.nat_only
in_port                 action  args
320      nat_only_port
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v4
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v6
swadm table dump pipe.Ingress.mcast_ingress.mcast_replication_ipv6
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv4
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4_mcast
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6_mcast
swadm table dump pipe.Egress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:35:02
428   rewrite  mac=a8:40:25:05:35:03
36    rewrite  mac=a8:40:25:05:35:04
424   rewrite  mac=a8:40:25:05:35:05
32    rewrite  mac=a8:40:25:05:35:06
444   rewrite  mac=a8:40:25:05:35:07
20    rewrite  mac=a8:40:25:05:35:08
440   rewrite  mac=a8:40:25:05:35:09
16    rewrite  mac=a8:40:25:05:35:0a
436   rewrite  mac=a8:40:25:05:35:0b
28    rewrite  mac=a8:40:25:05:35:0c
432   rewrite  mac=a8:40:25:05:35:0d
24    rewrite  mac=a8:40:25:05:35:0e
452   rewrite  mac=a8:40:25:05:35:0f
12    rewrite  mac=a8:40:25:05:35:10
448   rewrite  mac=a8:40:25:05:35:11
8     rewrite  mac=a8:40:25:05:35:12
416   rewrite  mac=a8:40:25:05:35:13
40    rewrite  mac=a8:40:25:05:35:14
420   rewrite  mac=a8:40:25:05:35:15
44    rewrite  mac=a8:40:25:05:35:16
400   rewrite  mac=a8:40:25:05:35:17
56    rewrite  mac=a8:40:25:05:35:18
404   rewrite  mac=a8:40:25:05:35:19
60    rewrite  mac=a8:40:25:05:35:1a
408   rewrite  mac=a8:40:25:05:35:1b
48    rewrite  mac=a8:40:25:05:35:1c
412   rewrite  mac=a8:40:25:05:35:1d
52    rewrite  mac=a8:40:25:05:35:1e
392   rewrite  mac=a8:40:25:05:35:1f
64    rewrite  mac=a8:40:25:05:35:20
396   rewrite  mac=a8:40:25:05:35:21
68    rewrite  mac=a8:40:25:05:35:22
320   rewrite  mac=a8:40:25:05:35:23
swadm table dump pipe.Egress.mcast_egress.tbl_decap_ports
swadm table dump pipe.Egress.mcast_egress.asic_id_to_port
asic_port_id                   action  args
2             set_port_number  port_number=0
428           set_port_number  port_number=0
36            set_port_number  port_number=1
424           set_port_number  port_number=2
32            set_port_number  port_number=3
444           set_port_number  port_number=4
20            set_port_number  port_number=5
440           set_port_number  port_number=6
16            set_port_number  port_number=7
436           set_port_number  port_number=8
28            set_port_number  port_number=9
432           set_port_number  port_number=10
24            set_port_number  port_number=11
452           set_port_number  port_number=12
12            set_port_number  port_number=13
448           set_port_number  port_number=14
8             set_port_number  port_number=15
416           set_port_number  port_number=16
40            set_port_number  port_number=17
420           set_port_number  port_number=18
44            set_port_number  port_number=19
400           set_port_number  port_number=20
56            set_port_number  port_number=21
404           set_port_number  port_number=22
60            set_port_number  port_number=23
408           set_port_number  port_number=24
48            set_port_number  port_number=25
412           set_port_number  port_number=26
52            set_port_number  port_number=27
392           set_port_number  port_number=28
64            set_port_number  port_number=29
396           set_port_number  port_number=30
68            set_port_number  port_number=31
320           set_port_number  port_number=0
root@oxz_switch0:~#

[rcgoodfellow]

Something very interesting that @JustinAzoff noticed.

When we look at the nat ingress path we see

root@oxz_switch0:~# swadm nat ls
External IP  Port low  Port high  Internal IP            Inner MAC          VNI
172.20.30.5  0         16383      fde4:f501:6a7d:101::1  a8:40:25:ff:c5:31  100
172.20.30.5  16384     32767      fde4:f501:6a7d:102::1  a8:40:25:ff:d0:f3  100

In this case I am running a ping at 10-15 packets a second and we see corresponding bumps in the nat ingress counter.

root@oxz_switch0:~# swadm table counters pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports        Packets  Bytes
172.20.30.5  0-16383      251420   25719494
172.20.30.5  16384-32767  1609     237164
root@oxz_switch0:~# swadm table counters pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports        Packets  Bytes
172.20.30.5  0-16383      251452   25722758
172.20.30.5  16384-32767  1609     237164
root@oxz_switch0:~# swadm table counters pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports        Packets  Bytes
172.20.30.5  0-16383      251519   25729592
172.20.30.5  16384-32767  1609     237164

that should lead to corresponding bumps in the ipv6 lookup table for the nat target, but it does not, there is only constant light traffic, which is likely ddm underlay heartbeats.

root@oxz_switch0:~# swadm table counters pipe.Ingress.l3_router.Router6.lookup_idx.lookup | grep fde4:f501:6a7d:101
fde4:f501:6a7d:101::/64  61740    18499324
root@oxz_switch0:~# swadm table counters pipe.Ingress.l3_router.Router6.lookup_idx.lookup | grep fde4:f501:6a7d:101
fde4:f501:6a7d:101::/64  61743    18499762
root@oxz_switch0:~# swadm table counters pipe.Ingress.l3_router.Router6.lookup_idx.lookup | grep fde4:f501:6a7d:101
fde4:f501:6a7d:101::/64  61755    18502874

So this puts us in the very odd position where either

a) We are not getting from nat encapsulation to ipv6 route lookups
b) Ipv6 route lookups are working on the underlay but not for nat encapsulated packets

This is even stranger given the fact that we've observed london exhibiting this pathology and then packets suddenly start flowing as expected after 12 hours.

update: on a system where nat is working with the current dendrite code, the counters for pipe.Ingress.l3_router.Router6.lookup_idx.lookup do increase at a rate that corresponds to the nat counters.

[rcgoodfellow]

On a recent dublin run, we've found that sw0 does not forward nat but sw1 does. Here are the tables from the switches.

sw0

root@oxz_switch0:~# swadm table list | xargs -t -l swadm table dump
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.lookup
dst_addr          action  args
0.0.0.0/0  index  idx=0   slots=1
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.route
idx           action                args
0    forward  nexthop=172.20.15.57  port=320
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.lookup
dst_addr                        action  args
fdf1:eb31:4d93:104::/64  index  idx=0   slots=1
fdf1:eb31:4d93:102::/64  index  idx=1   slots=1
fdb0:a840:2504:851::/64  index  idx=2   slots=1
fdf1:eb31:4d93:2::/64    index  idx=3   slots=1
fdb0:a840:2504:614::/64  index  idx=4   slots=1
fdf1:eb31:4d93:1::/64    index  idx=5   slots=1
fdb0:a840:2504:351::/64  index  idx=6   slots=1
fdf1:eb31:4d93:101::/64  index  idx=7   slots=1
fdb0:a840:2504:6d3::/64  index  idx=8   slots=1
fdf1:eb31:4d93:103::/64  index  idx=9   slots=1
fdf1:eb31:4d93:3::/64    index  idx=10  slots=1
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.route
idx           action                            args
0    forward  nexthop=fe80::aa40:25ff:fe04:851  port=40
1    forward  nexthop=fe80::aa40:25ff:fe04:614  port=8
2    forward  nexthop=fe80::aa40:25ff:fe04:851  port=40
3    forward  nexthop=fe80::aa40:25ff:fe04:614  port=8
4    forward  nexthop=fe80::aa40:25ff:fe04:614  port=8
5    forward  nexthop=fe80::aa40:25ff:fe04:351  port=448
6    forward  nexthop=fe80::aa40:25ff:fe04:351  port=448
7    forward  nexthop=fe80::aa40:25ff:fe04:351  port=448
8    forward  nexthop=fe80::aa40:25ff:fe04:6d3  port=416
9    forward  nexthop=fe80::aa40:25ff:fe04:6d3  port=416
10   forward  nexthop=fe80::aa40:25ff:fe04:6d3  port=416
swadm table dump pipe.Ingress.l3_router.MulticastRouter4.tbl
swadm table dump pipe.Ingress.l3_router.MulticastRouter6.tbl
swadm table dump pipe.Ingress.l3_router.Arp.tbl
ip                     action  args
172.20.15.61  rewrite  dst_mac=a8:40:25:05:22:23
172.20.15.57  rewrite  dst_mac=aa:00:04:00:ca:fe
172.20.15.59  rewrite  dst_mac=aa:00:04:00:ca:fe
swadm table dump pipe.Ingress.l3_router.Ndp.tbl
ip                                  action  args
fe80::aa40:25ff:fe05:2203  rewrite  dst_mac=a8:40:25:05:22:03
fe80::aa40:25ff:fe05:2202  rewrite  dst_mac=a8:40:25:05:22:02
fe80::aa40:25ff:fe05:220e  rewrite  dst_mac=a8:40:25:05:22:0e
fe80::aa40:25ff:fe05:220d  rewrite  dst_mac=a8:40:25:05:22:0d
fe80::aa40:25ff:fe05:2217  rewrite  dst_mac=a8:40:25:05:22:17
fe80::aa40:25ff:fe05:2216  rewrite  dst_mac=a8:40:25:05:22:16
fe80::aa40:25ff:fe05:2215  rewrite  dst_mac=a8:40:25:05:22:15
fe80::aa40:25ff:fe05:2214  rewrite  dst_mac=a8:40:25:05:22:14
fe80::aa40:25ff:fe05:2213  rewrite  dst_mac=a8:40:25:05:22:13
fe80::aa40:25ff:fe05:2212  rewrite  dst_mac=a8:40:25:05:22:12
fe80::aa40:25ff:fe05:2211  rewrite  dst_mac=a8:40:25:05:22:11
fe80::aa40:25ff:fe05:2210  rewrite  dst_mac=a8:40:25:05:22:10
fe80::aa40:25ff:fe05:221f  rewrite  dst_mac=a8:40:25:05:22:1f
fe80::aa40:25ff:fe05:221e  rewrite  dst_mac=a8:40:25:05:22:1e
fe80::aa40:25ff:fe05:221d  rewrite  dst_mac=a8:40:25:05:22:1d
fe80::aa40:25ff:fe05:221c  rewrite  dst_mac=a8:40:25:05:22:1c
fe80::aa40:25ff:fe05:221b  rewrite  dst_mac=a8:40:25:05:22:1b
fe80::aa40:25ff:fe05:221a  rewrite  dst_mac=a8:40:25:05:22:1a
fe80::aa40:25ff:fe05:2219  rewrite  dst_mac=a8:40:25:05:22:19
fe80::aa40:25ff:fe05:2218  rewrite  dst_mac=a8:40:25:05:22:18
fe80::aa40:25ff:fe05:2207  rewrite  dst_mac=a8:40:25:05:22:07
fe80::aa40:25ff:fe05:2206  rewrite  dst_mac=a8:40:25:05:22:06
fe80::aa40:25ff:fe05:2205  rewrite  dst_mac=a8:40:25:05:22:05
fe80::aa40:25ff:fe05:2204  rewrite  dst_mac=a8:40:25:05:22:04
fe80::aa40:25ff:fe05:220f  rewrite  dst_mac=a8:40:25:05:22:0f
fe80::aa40:25ff:fe05:2222  rewrite  dst_mac=a8:40:25:05:22:22
fe80::aa40:25ff:fe05:2221  rewrite  dst_mac=a8:40:25:05:22:21
fe80::aa40:25ff:fe05:2220  rewrite  dst_mac=a8:40:25:05:22:20
fe80::aa40:25ff:fe05:220c  rewrite  dst_mac=a8:40:25:05:22:0c
fe80::aa40:25ff:fe05:220b  rewrite  dst_mac=a8:40:25:05:22:0b
fe80::aa40:25ff:fe05:220a  rewrite  dst_mac=a8:40:25:05:22:0a
fe80::aa40:25ff:fe05:2209  rewrite  dst_mac=a8:40:25:05:22:09
fe80::aa40:25ff:fe05:2208  rewrite  dst_mac=a8:40:25:05:22:08
fe80::aa40:25ff:fe04:614   rewrite  dst_mac=a8:40:25:04:06:14
fe80::aa40:25ff:fe04:351   rewrite  dst_mac=a8:40:25:04:03:51
fe80::aa40:25ff:fe04:851   rewrite  dst_mac=a8:40:25:04:08:51
fe80::aa40:25ff:fe04:6d3   rewrite  dst_mac=a8:40:25:04:06:d3
swadm table dump pipe.Ingress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:22:02
428   rewrite  mac=a8:40:25:05:22:03
36    rewrite  mac=a8:40:25:05:22:04
424   rewrite  mac=a8:40:25:05:22:05
32    rewrite  mac=a8:40:25:05:22:06
444   rewrite  mac=a8:40:25:05:22:07
20    rewrite  mac=a8:40:25:05:22:08
440   rewrite  mac=a8:40:25:05:22:09
16    rewrite  mac=a8:40:25:05:22:0a
436   rewrite  mac=a8:40:25:05:22:0b
28    rewrite  mac=a8:40:25:05:22:0c
432   rewrite  mac=a8:40:25:05:22:0d
24    rewrite  mac=a8:40:25:05:22:0e
452   rewrite  mac=a8:40:25:05:22:0f
12    rewrite  mac=a8:40:25:05:22:10
448   rewrite  mac=a8:40:25:05:22:11
8     rewrite  mac=a8:40:25:05:22:12
416   rewrite  mac=a8:40:25:05:22:13
40    rewrite  mac=a8:40:25:05:22:14
420   rewrite  mac=a8:40:25:05:22:15
44    rewrite  mac=a8:40:25:05:22:16
400   rewrite  mac=a8:40:25:05:22:17
56    rewrite  mac=a8:40:25:05:22:18
404   rewrite  mac=a8:40:25:05:22:19
60    rewrite  mac=a8:40:25:05:22:1a
408   rewrite  mac=a8:40:25:05:22:1b
48    rewrite  mac=a8:40:25:05:22:1c
412   rewrite  mac=a8:40:25:05:22:1d
52    rewrite  mac=a8:40:25:05:22:1e
392   rewrite  mac=a8:40:25:05:22:1f
64    rewrite  mac=a8:40:25:05:22:20
396   rewrite  mac=a8:40:25:05:22:21
68    rewrite  mac=a8:40:25:05:22:22
320   rewrite  mac=a8:40:25:05:22:23
swadm table dump pipe.Ingress.filter.switch_ipv4_addr
dst_addr      in_port               action  args
172.20.15.61  0x140/0x1ff  claimv4
172.20.15.61  0x0/0x0      dropv4
swadm table dump pipe.Ingress.filter.switch_ipv6_addr
dst_addr                   in_port               action  args
fdd4:9500:e894:986c::1     0x0/0x0      claimv6
fdb1:a840:2504:351::1      0x2/0x1ff    claimv6
fdb1:a840:2504:351::1      0x0/0x0      dropv6
fdb2:a840:2504:351::1      0x2/0x1ff    claimv6
fdb2:a840:2504:351::1      0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2202  0x2/0x1ff    claimv6
fe80::aa40:25ff:fe05:2202  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2203  0x1ac/0x1ff  claimv6
fe80::aa40:25ff:fe05:2203  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220d  0x1b0/0x1ff  claimv6
fe80::aa40:25ff:fe05:220d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220e  0x18/0x1ff   claimv6
fe80::aa40:25ff:fe05:220e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220f  0x1c4/0x1ff  claimv6
fe80::aa40:25ff:fe05:220f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2210  0xc/0x1ff    claimv6
fe80::aa40:25ff:fe05:2210  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2211  0x1c0/0x1ff  claimv6
fe80::aa40:25ff:fe05:2211  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2212  0x8/0x1ff    claimv6
fe80::aa40:25ff:fe05:2212  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2213  0x1a0/0x1ff  claimv6
fe80::aa40:25ff:fe05:2213  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2214  0x28/0x1ff   claimv6
fe80::aa40:25ff:fe05:2214  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2215  0x1a4/0x1ff  claimv6
fe80::aa40:25ff:fe05:2215  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2216  0x2c/0x1ff   claimv6
fe80::aa40:25ff:fe05:2216  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2204  0x24/0x1ff   claimv6
fe80::aa40:25ff:fe05:2204  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2217  0x190/0x1ff  claimv6
fe80::aa40:25ff:fe05:2217  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2218  0x38/0x1ff   claimv6
fe80::aa40:25ff:fe05:2218  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2219  0x194/0x1ff  claimv6
fe80::aa40:25ff:fe05:2219  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221a  0x3c/0x1ff   claimv6
fe80::aa40:25ff:fe05:221a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221b  0x198/0x1ff  claimv6
fe80::aa40:25ff:fe05:221b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221c  0x30/0x1ff   claimv6
fe80::aa40:25ff:fe05:221c  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221d  0x19c/0x1ff  claimv6
fe80::aa40:25ff:fe05:221d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221e  0x34/0x1ff   claimv6
fe80::aa40:25ff:fe05:221e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:221f  0x188/0x1ff  claimv6
fe80::aa40:25ff:fe05:221f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2220  0x40/0x1ff   claimv6
fe80::aa40:25ff:fe05:2220  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2205  0x1a8/0x1ff  claimv6
fe80::aa40:25ff:fe05:2205  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2221  0x18c/0x1ff  claimv6
fe80::aa40:25ff:fe05:2221  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2222  0x44/0x1ff   claimv6
fe80::aa40:25ff:fe05:2222  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2206  0x20/0x1ff   claimv6
fe80::aa40:25ff:fe05:2206  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2207  0x1bc/0x1ff  claimv6
fe80::aa40:25ff:fe05:2207  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2208  0x14/0x1ff   claimv6
fe80::aa40:25ff:fe05:2208  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2209  0x1b8/0x1ff  claimv6
fe80::aa40:25ff:fe05:2209  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220a  0x10/0x1ff   claimv6
fe80::aa40:25ff:fe05:220a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220b  0x1b4/0x1ff  claimv6
fe80::aa40:25ff:fe05:220b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:220c  0x1c/0x1ff   claimv6
fe80::aa40:25ff:fe05:220c  0x0/0x0      dropv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports                     action                       args
172.20.29.5  0-16383  forward_ipv4_to  inner_mac=a8:40:25:ff:c6:69  target=fdf1:eb31:4d93:101::1  vni=100
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6
swadm table dump pipe.Ingress.nat_ingress.nat_only
in_port                 action  args
320      nat_only_port
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v4
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v6
swadm table dump pipe.Ingress.mcast_ingress.mcast_replication_ipv6
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv4
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4_mcast
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6_mcast
swadm table dump pipe.Egress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:22:02
428   rewrite  mac=a8:40:25:05:22:03
36    rewrite  mac=a8:40:25:05:22:04
424   rewrite  mac=a8:40:25:05:22:05
32    rewrite  mac=a8:40:25:05:22:06
444   rewrite  mac=a8:40:25:05:22:07
20    rewrite  mac=a8:40:25:05:22:08
440   rewrite  mac=a8:40:25:05:22:09
16    rewrite  mac=a8:40:25:05:22:0a
436   rewrite  mac=a8:40:25:05:22:0b
28    rewrite  mac=a8:40:25:05:22:0c
432   rewrite  mac=a8:40:25:05:22:0d
24    rewrite  mac=a8:40:25:05:22:0e
452   rewrite  mac=a8:40:25:05:22:0f
12    rewrite  mac=a8:40:25:05:22:10
448   rewrite  mac=a8:40:25:05:22:11
8     rewrite  mac=a8:40:25:05:22:12
416   rewrite  mac=a8:40:25:05:22:13
40    rewrite  mac=a8:40:25:05:22:14
420   rewrite  mac=a8:40:25:05:22:15
44    rewrite  mac=a8:40:25:05:22:16
400   rewrite  mac=a8:40:25:05:22:17
56    rewrite  mac=a8:40:25:05:22:18
404   rewrite  mac=a8:40:25:05:22:19
60    rewrite  mac=a8:40:25:05:22:1a
408   rewrite  mac=a8:40:25:05:22:1b
48    rewrite  mac=a8:40:25:05:22:1c
412   rewrite  mac=a8:40:25:05:22:1d
52    rewrite  mac=a8:40:25:05:22:1e
392   rewrite  mac=a8:40:25:05:22:1f
64    rewrite  mac=a8:40:25:05:22:20
396   rewrite  mac=a8:40:25:05:22:21
68    rewrite  mac=a8:40:25:05:22:22
320   rewrite  mac=a8:40:25:05:22:23
swadm table dump pipe.Egress.mcast_egress.tbl_decap_ports
swadm table dump pipe.Egress.mcast_egress.asic_id_to_port
asic_port_id                   action  args
2             set_port_number  port_number=0
428           set_port_number  port_number=0
36            set_port_number  port_number=1
424           set_port_number  port_number=2
32            set_port_number  port_number=3
444           set_port_number  port_number=4
20            set_port_number  port_number=5
440           set_port_number  port_number=6
16            set_port_number  port_number=7
436           set_port_number  port_number=8
28            set_port_number  port_number=9
432           set_port_number  port_number=10
24            set_port_number  port_number=11
452           set_port_number  port_number=12
12            set_port_number  port_number=13
448           set_port_number  port_number=14
8             set_port_number  port_number=15
416           set_port_number  port_number=16
40            set_port_number  port_number=17
420           set_port_number  port_number=18
44            set_port_number  port_number=19
400           set_port_number  port_number=20
56            set_port_number  port_number=21
404           set_port_number  port_number=22
60            set_port_number  port_number=23
408           set_port_number  port_number=24
48            set_port_number  port_number=25
412           set_port_number  port_number=26
52            set_port_number  port_number=27
392           set_port_number  port_number=28
64            set_port_number  port_number=29
396           set_port_number  port_number=30
68            set_port_number  port_number=31
320           set_port_number  port_number=0

sw1

root@oxz_switch1:~# swadm table list | xargs -t -l swadm table dump
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.lookup
dst_addr          action  args
0.0.0.0/0  index  idx=0   slots=1
swadm table dump pipe.Ingress.l3_router.Router4.lookup_idx.route
idx           action                args
0    forward  nexthop=172.20.15.57  port=320
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.lookup
dst_addr                        action  args
fdb0:a840:2504:6d3::/64  index  idx=0   slots=1
fdf1:eb31:4d93:103::/64  index  idx=1   slots=1
fdf1:eb31:4d93:2::/64    index  idx=2   slots=1
fdb0:a840:2504:851::/64  index  idx=3   slots=1
fdf1:eb31:4d93:102::/64  index  idx=4   slots=1
fdf1:eb31:4d93:104::/64  index  idx=5   slots=1
fdb0:a840:2504:614::/64  index  idx=6   slots=1
fdf1:eb31:4d93:101::/64  index  idx=7   slots=1
fdf1:eb31:4d93:1::/64    index  idx=8   slots=1
fdb0:a840:2504:351::/64  index  idx=9   slots=1
fdf1:eb31:4d93:3::/64    index  idx=10  slots=1
swadm table dump pipe.Ingress.l3_router.Router6.lookup_idx.route
idx           action                            args
0    forward  nexthop=fe80::aa40:25ff:fe04:6db  port=416
1    forward  nexthop=fe80::aa40:25ff:fe04:6db  port=416
2    forward  nexthop=fe80::aa40:25ff:fe04:61c  port=8
3    forward  nexthop=fe80::aa40:25ff:fe04:859  port=40
4    forward  nexthop=fe80::aa40:25ff:fe04:61c  port=8
5    forward  nexthop=fe80::aa40:25ff:fe04:859  port=40
6    forward  nexthop=fe80::aa40:25ff:fe04:61c  port=8
7    forward  nexthop=fe80::aa40:25ff:fe04:359  port=448
8    forward  nexthop=fe80::aa40:25ff:fe04:359  port=448
9    forward  nexthop=fe80::aa40:25ff:fe04:359  port=448
10   forward  nexthop=fe80::aa40:25ff:fe04:6db  port=416
swadm table dump pipe.Ingress.l3_router.MulticastRouter4.tbl
swadm table dump pipe.Ingress.l3_router.MulticastRouter6.tbl
swadm table dump pipe.Ingress.l3_router.Arp.tbl
ip                     action  args
172.20.15.62  rewrite  dst_mac=a8:40:25:05:2e:23
172.20.15.57  rewrite  dst_mac=aa:00:04:00:ca:fe
172.20.15.59  rewrite  dst_mac=aa:00:04:00:ca:fe
swadm table dump pipe.Ingress.l3_router.Ndp.tbl
ip                                  action  args
fe80::aa40:25ff:fe05:2e12  rewrite  dst_mac=a8:40:25:05:2e:12
fe80::aa40:25ff:fe05:2e11  rewrite  dst_mac=a8:40:25:05:2e:11
fe80::aa40:25ff:fe05:2e10  rewrite  dst_mac=a8:40:25:05:2e:10
fe80::aa40:25ff:fe05:2e0f  rewrite  dst_mac=a8:40:25:05:2e:0f
fe80::aa40:25ff:fe05:2e0e  rewrite  dst_mac=a8:40:25:05:2e:0e
fe80::aa40:25ff:fe05:2e0d  rewrite  dst_mac=a8:40:25:05:2e:0d
fe80::aa40:25ff:fe05:2e03  rewrite  dst_mac=a8:40:25:05:2e:03
fe80::aa40:25ff:fe05:2e02  rewrite  dst_mac=a8:40:25:05:2e:02
fe80::aa40:25ff:fe05:2e1b  rewrite  dst_mac=a8:40:25:05:2e:1b
fe80::aa40:25ff:fe05:2e1a  rewrite  dst_mac=a8:40:25:05:2e:1a
fe80::aa40:25ff:fe05:2e19  rewrite  dst_mac=a8:40:25:05:2e:19
fe80::aa40:25ff:fe05:2e18  rewrite  dst_mac=a8:40:25:05:2e:18
fe80::aa40:25ff:fe05:2e1f  rewrite  dst_mac=a8:40:25:05:2e:1f
fe80::aa40:25ff:fe05:2e1e  rewrite  dst_mac=a8:40:25:05:2e:1e
fe80::aa40:25ff:fe05:2e1d  rewrite  dst_mac=a8:40:25:05:2e:1d
fe80::aa40:25ff:fe05:2e1c  rewrite  dst_mac=a8:40:25:05:2e:1c
fe80::aa40:25ff:fe05:2e13  rewrite  dst_mac=a8:40:25:05:2e:13
fe80::aa40:25ff:fe05:2e17  rewrite  dst_mac=a8:40:25:05:2e:17
fe80::aa40:25ff:fe05:2e16  rewrite  dst_mac=a8:40:25:05:2e:16
fe80::aa40:25ff:fe05:2e15  rewrite  dst_mac=a8:40:25:05:2e:15
fe80::aa40:25ff:fe05:2e14  rewrite  dst_mac=a8:40:25:05:2e:14
fe80::aa40:25ff:fe05:2e0a  rewrite  dst_mac=a8:40:25:05:2e:0a
fe80::aa40:25ff:fe05:2e09  rewrite  dst_mac=a8:40:25:05:2e:09
fe80::aa40:25ff:fe05:2e08  rewrite  dst_mac=a8:40:25:05:2e:08
fe80::aa40:25ff:fe05:2e07  rewrite  dst_mac=a8:40:25:05:2e:07
fe80::aa40:25ff:fe05:2e06  rewrite  dst_mac=a8:40:25:05:2e:06
fe80::aa40:25ff:fe05:2e05  rewrite  dst_mac=a8:40:25:05:2e:05
fe80::aa40:25ff:fe05:2e04  rewrite  dst_mac=a8:40:25:05:2e:04
fe80::aa40:25ff:fe05:2e22  rewrite  dst_mac=a8:40:25:05:2e:22
fe80::aa40:25ff:fe05:2e21  rewrite  dst_mac=a8:40:25:05:2e:21
fe80::aa40:25ff:fe05:2e20  rewrite  dst_mac=a8:40:25:05:2e:20
fe80::aa40:25ff:fe05:2e0b  rewrite  dst_mac=a8:40:25:05:2e:0b
fe80::aa40:25ff:fe05:2e0c  rewrite  dst_mac=a8:40:25:05:2e:0c
fe80::aa40:25ff:fe04:61c   rewrite  dst_mac=a8:40:25:04:06:1c
fe80::aa40:25ff:fe04:859   rewrite  dst_mac=a8:40:25:04:08:59
fe80::aa40:25ff:fe04:359   rewrite  dst_mac=a8:40:25:04:03:59
fe80::aa40:25ff:fe04:6db   rewrite  dst_mac=a8:40:25:04:06:db
swadm table dump pipe.Ingress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:2e:02
428   rewrite  mac=a8:40:25:05:2e:03
36    rewrite  mac=a8:40:25:05:2e:04
424   rewrite  mac=a8:40:25:05:2e:05
32    rewrite  mac=a8:40:25:05:2e:06
444   rewrite  mac=a8:40:25:05:2e:07
20    rewrite  mac=a8:40:25:05:2e:08
440   rewrite  mac=a8:40:25:05:2e:09
16    rewrite  mac=a8:40:25:05:2e:0a
436   rewrite  mac=a8:40:25:05:2e:0b
28    rewrite  mac=a8:40:25:05:2e:0c
432   rewrite  mac=a8:40:25:05:2e:0d
24    rewrite  mac=a8:40:25:05:2e:0e
452   rewrite  mac=a8:40:25:05:2e:0f
12    rewrite  mac=a8:40:25:05:2e:10
448   rewrite  mac=a8:40:25:05:2e:11
8     rewrite  mac=a8:40:25:05:2e:12
416   rewrite  mac=a8:40:25:05:2e:13
40    rewrite  mac=a8:40:25:05:2e:14
420   rewrite  mac=a8:40:25:05:2e:15
44    rewrite  mac=a8:40:25:05:2e:16
400   rewrite  mac=a8:40:25:05:2e:17
56    rewrite  mac=a8:40:25:05:2e:18
404   rewrite  mac=a8:40:25:05:2e:19
60    rewrite  mac=a8:40:25:05:2e:1a
408   rewrite  mac=a8:40:25:05:2e:1b
48    rewrite  mac=a8:40:25:05:2e:1c
412   rewrite  mac=a8:40:25:05:2e:1d
52    rewrite  mac=a8:40:25:05:2e:1e
392   rewrite  mac=a8:40:25:05:2e:1f
64    rewrite  mac=a8:40:25:05:2e:20
396   rewrite  mac=a8:40:25:05:2e:21
68    rewrite  mac=a8:40:25:05:2e:22
320   rewrite  mac=a8:40:25:05:2e:23
swadm table dump pipe.Ingress.filter.switch_ipv4_addr
dst_addr      in_port               action  args
172.20.15.62  0x140/0x1ff  claimv4
172.20.15.62  0x0/0x0      dropv4
swadm table dump pipe.Ingress.filter.switch_ipv6_addr
dst_addr                   in_port               action  args
fdb1:a840:2504:6d3::1      0x2/0x1ff    claimv6
fdb1:a840:2504:6d3::1      0x0/0x0      dropv6
fdb2:a840:2504:6d3::1      0x2/0x1ff    claimv6
fdb2:a840:2504:6d3::1      0x0/0x0      dropv6
fdea:633a:5bf4:bd82::1     0x0/0x0      claimv6
fe80::aa40:25ff:fe05:2e02  0x2/0x1ff    claimv6
fe80::aa40:25ff:fe05:2e02  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e03  0x1ac/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e03  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0d  0x1b0/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e0d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0e  0x18/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e0e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0f  0x1c4/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e0f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e10  0xc/0x1ff    claimv6
fe80::aa40:25ff:fe05:2e10  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e11  0x1c0/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e11  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e12  0x8/0x1ff    claimv6
fe80::aa40:25ff:fe05:2e12  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e13  0x1a0/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e13  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e14  0x28/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e14  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e15  0x1a4/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e15  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e16  0x2c/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e16  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e04  0x24/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e04  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e17  0x190/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e17  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e18  0x38/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e18  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e19  0x194/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e19  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1a  0x3c/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e1a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1b  0x198/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e1b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1c  0x30/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e1c  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1d  0x19c/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e1d  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1e  0x34/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e1e  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e1f  0x188/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e1f  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e20  0x40/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e20  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e05  0x1a8/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e05  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e21  0x18c/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e21  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e22  0x44/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e22  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e06  0x20/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e06  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e07  0x1bc/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e07  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e08  0x14/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e08  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e09  0x1b8/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e09  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0a  0x10/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e0a  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0b  0x1b4/0x1ff  claimv6
fe80::aa40:25ff:fe05:2e0b  0x0/0x0      dropv6
fe80::aa40:25ff:fe05:2e0c  0x1c/0x1ff   claimv6
fe80::aa40:25ff:fe05:2e0c  0x0/0x0      dropv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4
dst_addr     ports                     action                       args
172.20.29.5  0-16383  forward_ipv4_to  inner_mac=a8:40:25:ff:c6:69  target=fdf1:eb31:4d93:101::1  vni=100
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6
swadm table dump pipe.Ingress.nat_ingress.nat_only
in_port                 action  args
320      nat_only_port
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v4
swadm table dump pipe.Ingress.attached_subnet_ingress.attached_subnets_v6
swadm table dump pipe.Ingress.mcast_ingress.mcast_replication_ipv6
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv4
swadm table dump pipe.Ingress.mcast_ingress.mcast_source_filter_ipv6
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv4_mcast
swadm table dump pipe.Ingress.nat_ingress.ingress_ipv6_mcast
swadm table dump pipe.Egress.mac_rewrite.mac_rewrite
port           action  args
2     rewrite  mac=a8:40:25:05:2e:02
428   rewrite  mac=a8:40:25:05:2e:03
36    rewrite  mac=a8:40:25:05:2e:04
424   rewrite  mac=a8:40:25:05:2e:05
32    rewrite  mac=a8:40:25:05:2e:06
444   rewrite  mac=a8:40:25:05:2e:07
20    rewrite  mac=a8:40:25:05:2e:08
440   rewrite  mac=a8:40:25:05:2e:09
16    rewrite  mac=a8:40:25:05:2e:0a
436   rewrite  mac=a8:40:25:05:2e:0b
28    rewrite  mac=a8:40:25:05:2e:0c
432   rewrite  mac=a8:40:25:05:2e:0d
24    rewrite  mac=a8:40:25:05:2e:0e
452   rewrite  mac=a8:40:25:05:2e:0f
12    rewrite  mac=a8:40:25:05:2e:10
448   rewrite  mac=a8:40:25:05:2e:11
8     rewrite  mac=a8:40:25:05:2e:12
416   rewrite  mac=a8:40:25:05:2e:13
40    rewrite  mac=a8:40:25:05:2e:14
420   rewrite  mac=a8:40:25:05:2e:15
44    rewrite  mac=a8:40:25:05:2e:16
400   rewrite  mac=a8:40:25:05:2e:17
56    rewrite  mac=a8:40:25:05:2e:18
404   rewrite  mac=a8:40:25:05:2e:19
60    rewrite  mac=a8:40:25:05:2e:1a
408   rewrite  mac=a8:40:25:05:2e:1b
48    rewrite  mac=a8:40:25:05:2e:1c
412   rewrite  mac=a8:40:25:05:2e:1d
52    rewrite  mac=a8:40:25:05:2e:1e
392   rewrite  mac=a8:40:25:05:2e:1f
64    rewrite  mac=a8:40:25:05:2e:20
396   rewrite  mac=a8:40:25:05:2e:21
68    rewrite  mac=a8:40:25:05:2e:22
320   rewrite  mac=a8:40:25:05:2e:23
swadm table dump pipe.Egress.mcast_egress.tbl_decap_ports
swadm table dump pipe.Egress.mcast_egress.asic_id_to_port
asic_port_id                   action  args
2             set_port_number  port_number=0
428           set_port_number  port_number=0
36            set_port_number  port_number=1
424           set_port_number  port_number=2
32            set_port_number  port_number=3
444           set_port_number  port_number=4
20            set_port_number  port_number=5
440           set_port_number  port_number=6
16            set_port_number  port_number=7
436           set_port_number  port_number=8
28            set_port_number  port_number=9
432           set_port_number  port_number=10
24            set_port_number  port_number=11
452           set_port_number  port_number=12
12            set_port_number  port_number=13
448           set_port_number  port_number=14
8             set_port_number  port_number=15
416           set_port_number  port_number=16
40            set_port_number  port_number=17
420           set_port_number  port_number=18
44            set_port_number  port_number=19
400           set_port_number  port_number=20
56            set_port_number  port_number=21
404           set_port_number  port_number=22
60            set_port_number  port_number=23
408           set_port_number  port_number=24
48            set_port_number  port_number=25
412           set_port_number  port_number=26
52            set_port_number  port_number=27
392           set_port_number  port_number=28
64            set_port_number  port_number=29
396           set_port_number  port_number=30
68            set_port_number  port_number=31
320           set_port_number  port_number=0

[rcgoodfellow]

I've added a simulator test that tries to get as close as possible to the NAT issue from the dublin/sw0 table dumps above. Unfortunately this test passes and the simulator trace looks like what we expect the code to do when its working. Simulator trace attached.

model_20260125_075042.log

[rcgoodfellow]

I've added a default route to the v6 lookup table on berlin/sw0. That way even if the v6 target was getting corrupted post NAT, we'd see counters bump for the ::/0 route. We do not see LPM counters bumping for ::/0.

I also shut down all v4 traffic to berlin/sw0 except for the NAT stream that is failing. Since this is traffic that is originated from a rear port that we expect a response on, if we were incorrectly decapsulating packets we'd hit the default route entry for v4. In that case we'd expect double the packet rate on the v4 default route: one hit for going out and one hit coming back and getting decapsulated. We do not see that, so this strongly suggests that we are not making it to either the v6 lookup or the v4 lookup at all since they both have default routes now and we don't see a corresponding bump in counters.

[rcgoodfellow]

Update: @Nieuwejaar removed multicast support from the P4 codebase and we do not encounter NAT forwarding issues. That code should functionally not be on the scene at all along these data paths, further pointing toward a compiler issue.