r/networking • u/KaleidoscopeNo9726 • 11h ago
Troubleshooting A few seconds of downtime
I inherited a VXLAN EVPN network and it has been pretty stable until recently. From time to time, the layer3 or at least the inter-vlan drops for a few seconds. It happened last week, yesterday and today.
For what I can tell, the layer2 is fine. The VLANs gateway is on the service leafs (vPC pair). We are using ePBR to force the inter-vlan to the firewall, which is connected to the service leafs, the firewall will route the traffic back to the service leafs.
This has been stable for several months and no network changes. At this point, I'm not sure if the Nexus pair (where the SVI with ePBR) is causing the network downtime or the firewall.
We recently migrated to OpenShift for virtualization and containers. This was a month or two ago.
The firewall has a static route to the 172.16.0.0/16 with the next-hop of the HSRP VIP of the vPC pair. The service leafs are learning the 172.16/16 from a Catalyst leaf. All the VLANs are stretched to the service leafs via L2VNI.
```
[Fw]---/29---[service leafs pair]---[spines]---[cat9k leaf]
```
I know the layer2 (or L2VNI) didn't fail because Zabbix is not reporting any ICMPloss between it and the hosts in the same VLAN. However, any subnets within the same VRF drops for a few seconds.
The leafs CPU utilization history is basically idle 1% - 3%. The firewall is the same thing. Network utilization wise, we are using less than 800Mbps of traffic and the network is 40Gbps links.
I'm trying to get some ideas what I should look for to identity the issue and what is causing it.
My nexus version is 10.5.4 and the Catalyst is 17.15.4b. The Palo Alto firewalls (active/passive) are version 11.14.0.
Thanks.