A server or a downstream switch connected to two leaves needs both leaves to act as one. There are two ways to do that on a VXLAN EVPN fabric:
- An MLAG pair. On Nexus that's vPC: two switches joined by a peer link and a keepalive, presenting one LACP system to whatever is below them.
- EVPN multihoming. The two links are one ethernet segment, named by an ESI. The leaves find each other through BGP, elect a designated forwarder per segment, and need no peer link at all.
The second one is the better idea. It's a standard (RFC 7432) rather than a vendor feature, it scales past two switches, it gets rid of the peer link and all its failure modes, and it works the same way on every platform that implements it. If you're starting a fabric from nothing, it's what you'd pick.
So why isn't it offered on Nexus?
Because on NX-OS, the EVPN multihoming we'd need to generate arrives in release 10.6(1). That's recent enough that a fabric being built today can't count on running it, and a configuration for a release the switch doesn't run is worse than none. It either fails to load, or it loads and forwards differently from what the design shows.
So an L2 link with a Nexus at one end offers exactly three choices:
- None: a single link.
- Port-channel: several links to one switch.
- vPC: links to a pair of switches, bundled at the far end.
ESI stays on offer where the templates really build an ethernet segment: a Catalyst-only link on a VXLAN fabric, and an IOS-XR link on MPLS.
How this was found. For a while, ESI was on offer for Nexus links. The shared ESI and the bundle ID were both computed correctly. But the NX-OS templates never wrote an ethernet segment, so choosing ESI gave you a port-channel with nothing under it, and no warning. Removing the option was the fix. A project saved before the change that still asks for ESI on a Nexus link now gets a warning when it's generated.
What a vPC pair gets
vPC on a VXLAN leaf pair is more than a vpc domain. This is the core of what Design writes for the primary of a pair:
vpc domain 10
peer-switch
role priority 4096
peer-keepalive destination 169.254.12.2 source 169.254.12.1 vrf VPC-PKA
system-priority 8192
delay restore 300
auto-recovery
fast-convergence
peer-gateway
delay restore interface-vlan 30
ip arp synchronize
A few of those lines matter more than they look:
- The keepalive runs in its own VRF on a dedicated /30 from link-local space, so it never depends on the fabric it's there to protect.
peer-gatewaylets each switch route packets sent to its peer's MAC. Without it, some hosts' traffic crosses the peer link for no reason.delay restorekeeps the vPC ports and SVIs down after a reload until routing has converged, so the switch doesn't take in traffic it can't forward yet.
On the VXLAN side, the pair shares an anycast VTEP: the NVE interface is sourced from its own loopback with an anycast loopback beside it, so remote leaves see one next hop for anything behind the pair. Two more settings keep that honest:
interface nve1
advertise virtual-rmac
host-reachability protocol bgp
source-interface Loopback1 anycast Loopback2
advertise virtual-rmacgives both chassis the same router MAC for routes sourced from the anycast address. That's the pair's layer 3 identity, and it's a different thing from the vPC system MAC, which is its layer 2 identity.advertise-pip, on the BGP EVPN address family, advertises routes that only one member holds (a single-attached host, an external route) from that member's own address, so they aren't sent to the wrong half of the pair.
When this changes
When a fabric can assume NX-OS 10.6(1) or later, ESI on Nexus becomes worth building, and it'll be offered alongside vPC rather than instead of it. Until then, the tool only offers what the switch will actually run.
Questions or corrections: [email protected]. More about Design.