A server or a downstream switch connected to two leaves needs both leaves to act as one. There are two ways to do that on a VXLAN EVPN fabric:

  • An MLAG pair. On Nexus that's vPC: two switches joined by a peer link and a keepalive, presenting one LACP system to whatever is below them.
  • EVPN multihoming. The two links are one ethernet segment, named by an ESI. The leaves find each other through BGP, elect a designated forwarder per segment, and need no peer link at all.

The second one is the better idea. It's a standard (RFC 7432) rather than a vendor feature, it scales past two switches, it gets rid of the peer link and all its failure modes, and it works the same way on every platform that implements it. If you're starting a fabric from nothing, it's what you'd pick.

So why isn't it offered on Nexus?

Because on NX-OS, the EVPN multihoming we'd need to generate arrives in release 10.6(1). That's recent enough that a fabric being built today can't count on running it, and a configuration for a release the switch doesn't run is worse than none. It either fails to load, or it loads and forwards differently from what the design shows.

So an L2 link with a Nexus at one end offers exactly three choices:

  • None: a single link.
  • Port-channel: several links to one switch.
  • vPC: links to a pair of switches, bundled at the far end.

ESI stays on offer where the templates really build an ethernet segment: a Catalyst-only link on a VXLAN fabric, and an IOS-XR link on MPLS.

How this was found. For a while, ESI was on offer for Nexus links. The shared ESI and the bundle ID were both computed correctly. But the NX-OS templates never wrote an ethernet segment, so choosing ESI gave you a port-channel with nothing under it, and no warning. Removing the option was the fix. A project saved before the change that still asks for ESI on a Nexus link now gets a warning when it's generated.

What a vPC pair gets

vPC on a VXLAN leaf pair is more than a vpc domain. This is the core of what Design writes for the primary of a pair:

vpc domain 10
  peer-switch
  role priority 4096
  peer-keepalive destination 169.254.12.2 source 169.254.12.1 vrf VPC-PKA
  system-priority 8192
  delay restore 300
  auto-recovery
  fast-convergence
  peer-gateway
  delay restore interface-vlan 30
  ip arp synchronize

A few of those lines matter more than they look:

  • The keepalive runs in its own VRF on a dedicated /30 from link-local space, so it never depends on the fabric it's there to protect.
  • peer-gateway lets each switch route packets sent to its peer's MAC. Without it, some hosts' traffic crosses the peer link for no reason.
  • delay restore keeps the vPC ports and SVIs down after a reload until routing has converged, so the switch doesn't take in traffic it can't forward yet.

On the VXLAN side, the pair shares an anycast VTEP: the NVE interface is sourced from its own loopback with an anycast loopback beside it, so remote leaves see one next hop for anything behind the pair. Two more settings keep that honest:

interface nve1
  advertise virtual-rmac
  host-reachability protocol bgp
  source-interface Loopback1 anycast Loopback2
  • advertise virtual-rmac gives both chassis the same router MAC for routes sourced from the anycast address. That's the pair's layer 3 identity, and it's a different thing from the vPC system MAC, which is its layer 2 identity.
  • advertise-pip, on the BGP EVPN address family, advertises routes that only one member holds (a single-attached host, an external route) from that member's own address, so they aren't sent to the wrong half of the pair.

When this changes

When a fabric can assume NX-OS 10.6(1) or later, ESI on Nexus becomes worth building, and it'll be offered alongside vPC rather than instead of it. Until then, the tool only offers what the switch will actually run.

Questions or corrections: [email protected]. More about Design.