Tier-0 and Edge Node interfaces

Over the past few years I have deployed quite a few NSX-T infrastructures. One of the things that always causes some confusing (even for myself) is the following: what is the relationship between the Edge Node, the Tier-0, and their respective interfaces?

Edge Node

I’ll only be discussing the VM form factor Edge Nodes here.

The Edge Node is a VM that can be thought of as a box into which network services are deployed. By itself it doesn’t do anything, like an ESXi host without VMs. It behaves like any regular VM. It comes with a maximum of 4 (see edit below) interfaces, which can be configured as follows:

1. Eth0 – Management interface. This is what the NSX Manager uses to communicate with the Edge Node, and what you can use to SSH into it.
2. Fp-eth0 and;
3. Fp-eth1 – the traffic interfaces. These are used for VLAN and Overlay traffic. These are the ports that will be configured with the Tunnel End Points (TEP).
4. Fp-eth2 – another traffic interface, but this one is only used for bridging purposes. That’s a topic for another day.

Edit: Since version 3.2.1 the number of fp-eth interfaces has increased to 5.

These interfaces are also the interfaces you can see from vCenter when looking at the VM configuration.

In general; the interfaces connected to the Edge Node VM are listed as above. I say in general, because some deployments may vary slightly. The fp-eth0 and fp-eth1 interfaces can be connected to a trunk portgroup from vCenter or a trunk VLAN segment from NSX. This depends on whether the Edge Node TEPs and ESXi TEPs are configured in the same subnet or not – if they are the same an NSX segment is required, else a vCenter portgroup can be used.

These trunk interfaces provide the connectivity for all network traffic going through the Edge Node – Overlay and VLAN.

Tier-0 Gateway

The Tier-0 Gateway (or simply; T0), is a router instantiated on an Edge Node VM. Specifically the Service Router, or SR component “lives” there. We’ll ignore the Distributed Router, or DR component for this post.

The interfaces configured on the Tier-0 bind to the interfaces on the Edge Node VM – the fp-eth0 and fp-eth1 mentioned above.

These T0 interfaces are connected to uplink segments. In a default deployment this would be two separate VLANs, each designated to a separate upstream L3 device, such as top-of-rack switch. The reason for this is called ‘deterministic peering’ – a post about that is coming later.

From a logical perspective it looks as follows.

The T0 is located on the Edge Node VM, and its interfaces are connected to the uplink segments configured in NSX to a particular VLAN. This traffic is thus tagged on the segment, and tunneled through the trunks on the Edge Node VM, to the physical network. From there it reaches the upstream L3 device and routed to the rest of the infrastructure as necessary.

It’s all about the layers within NSX!

Recap

To summarize:

• The Edge Node VM has interfaces connected to either a management network (eth0) or a trunk (fp-eth0, fp-eth1, fp-eth2)
• The Tier-0 Gateway is deployed on the Edge Node VM
• The T0 interfaces are connected to uplink segments
• The uplink segments are trunked through the Edge Node VM interfaces