Understanding TLOCs With Cisco SD-WAN

I had a great conversation with someone recently about how to understand what TLOCs are in Viptela/Cisco SD-WAN, and a lot of it revolved around terminology. A lot of current routing terminology starts to break down or confuse the discussion about SD-WAN, and TLOCs are no exception as they have no analogue in traditional routing.

II’ll try to demystify TLOCs, and specifically how they pertain to the routing table in Cisco SD-WAN.

What’s a TLOC, Anyway?

A TLOC is a Transport Locator, which is analogous to a LISP RLOC, or Routing Locator. Both abstract the router itself in favor of location-based mapping. RLOCs represent the routing location and not an endpoint. This is different than traditional routing, where a router would be identified by its physical address on each interface.

How this applies to TLOCs, and Viptela specifically is that TLOCs are also a location-based mapping . A TLOC consists of a tuple of data:

• System-IP: This address is similar to an OSPF or BGP Router-ID and does not need to be routable or reachable.
• Transport Color: The color is used to differentiate different transports, ie, MPLS, Internet, LTE. In cases where transport types are duplicated but should still be treated differently from each other (such as two different Internet providers) the colors could be arbitrary, such as Silver and Gold. Part of this color information includes the details needed for data plane tunnel setup such as interface IPs, ports (for NAT-T) and some fabric information.
• Encapsulation Type: This is important for the advertising of data plane connectivity. The choices are IPSec or GRE, and for obvious reasons a GRE TLOC will not establish a data plane tunnel with an IPSec one.

Combining all three pieces of information creates a TLOC. An SD-WAN router with connectivity to three different transports (and colors) would have three TLOCs. Exchanging these TLOCs allows the arbitrary (or full-mesh) topology of the SD-WAN fabric to be built.

Put simply, TLOCs are data plane setup instructions sent by each WAN Edge to the vSmart. The vSmart then distributes the maps to the other WAN Edge routers per the topology policy, facilitating the data plane tunnel establishment.

How Do We Exchange TLOCs?

Take this simple topology as an example. In this example, each WAN Edge router has two TLOCs each. The System-IP and Encapsulation are the same for each TLOC, but the transport color is different (biz-internet and pub-internet).

Each WAN Edge router will advertise its two TLOCs to the vSmart, and, if there is no topology policy, the vSmart will just reflect the TLOCs to each other WAN Edge. The WAN Edge routers will attempt to establish data plane tunnels with every other TLOC from each of its own TLOCs.

Using traditional routing terminology, each physical interface of a router that is connected to a WAN transport will attempt to build a tunnel to every other router on each of its interfaces without the need to manually configure the IPSec/GRE parameters or specify sources and destinations.

What Does This Have To Do With Routing?

TLOCs serve another important function besides data plane connectivity. In OMP terms (the routing protocol used over the SD-WAN Fabric), the TLOC serves as a next-hop for route advertisements. OMP is very similar to BGP in many ways, and just as the next-hop must be resolvable for BGP to install a route, the same is true of OMP.

If a TLOC is filtered, any route advertisements which use that TLOC as a next-hop (via policy or other reason) will be flagged as invalid by the WAN Edge, and the route will not be installed in the RIB. This makes sense; after all, since a data plane tunnel can’t be established without the TLOC, why should the WAN Edge router consider the route valid for forwarding traffic? From the point of view of the WAN Edge router, the next-hop is unreachable.

If the next-hop TLOC is reachable, the route might be installed in the RIB assuming no other policy or preferences have been enforced. By default, Viptela uses Equal-Cost Multipathing on a per-flow basis. If a route is received with more than one next-hop TLOC, and those TLOCS are reachable, that TLOC is a candidate for load-balancing.

Why would a route have more than one next-hop TLOC? Remember that a TLOC is an abstraction, and a single WAN Edge router could have multiple TLOCs. In the above diagram, each route from a branch router will be advertised with two next-hop TLOCs as an example.

So One TLOC Equals One Tunnel?

This is where TLOCs get over-complicated, and it is a very common problem when trying to translate what we know of traditional routing into the concept of TLOCs.

Imagine a route advertised from the DC in the above diagram. In the RIB of a branch WAN Edge, the same route might have four different next-hop TLOCs, two TLOCs multiplied by two routers. Viptela uses up to four paths by default, though it can be configured to accept up to 16.

But wait, there’s more! By default, data plane tunnels can be established across different colors as long as there is reachability between those TLOCs. Let’s look again at the SD-WAN topology and try to show it in a way that makes more sense.

BR1-WAN1 is aware of a lot of TLOCs, but let’s narrow it down to a a single destination WAN Edge router for simplicity’s sake.

BR1-WAN1 is aware of two TLOCs for DC1-WAN1. Both TLOCs have the same system-ip and the same encapsulation, but the transport colors (and connection information associated) are different.Two TLOCs equals two tunnels, right? Wrong!

BR1-WAN1 will attempt to build a data plane tunnel to each of DC1-WAN1’s TLOCs over each its own TLOCs. Therefore, BR1-WAN1’s biz-internet TLOC will build a tunnel to DC1-WAN1’s biz-internet TLOC, and also to its pub-internet TLOC (through the TLOC extension to DC1-WAN2). BR1-WAN1 will do the same thing over its other TLOC using the pub-internet color. Assuming biz-internet and pub-internet have cross-connectivity somewhere within their respective clouds, that would be four data plane tunnels.

This is the magic of TLOCs! Scaling tunnels is accomplished easily because the solution takes care of tunnel establishment as a by-product of policy and WAN Edge router TLOC advertisements.

What does this mean for the routing table? In short, it means that TLOCs are the next-hops that matter, and not the particular data plane tunnel used to get there. By default, as long as data plane tunnels are up and equally weighted, each will be used and load balanced on a per-flow basis. Implementing more granular control over particular paths and tunnels requires a centralized policy.

Can You Show Me? My Head Hurts…

I think showing diagrams and text fails to properly explain how TLOCs matter in Viptela, so let’s examine the CLI of BR1-WAN1 directly. BR1-WAN1 is an IOS-XE SDWAN router with two TLOCs, one for biz-internet and one for pub-internet. That’s just the color name, in reality it could be Verizon Internet and Time Warner Internet or something like that.

The TLOC IPs do not directly correspond to the tunnels themselves, but instead are used as an identifier of the system-ip and color of the remote WAN Edge router. Each remote TLOC has four tunnels up as discussed before, despite there being only two colors due to the cross-color connectivity. On the inbound IPSec tunnels, you can see both the remote and local TLOC colors which should illustrate this.

So What Does the Routing Table Look Like?

Again, let’s take a look at the CLI of BR1-WAN1.

In IOS-XE SDWAN, the routing table itself only shows the valid next-hop as the System-IP of the advertising TLOC.

Despite there being four different data plane tunnels per next-hop in the routing table, for a total of eight possible paths (Though by default we will only use four) the RIB only shows two ‘next hop’ IPs. These next-hop IPs are actually the system-IPs of the remote WAN Edge routers that advertised the routes along with valid TLOCs.

The OMP Table must be consulted to validate whether a particular TLOC will be used for that prefix.

The OMP table shows all routes that were received from vSmarts as well as the route originator and TLOC, and whether or not the route is installed in the RIB.

In this lab, the DC WAN Edges are vEdges, and the Branch WAN Edges are IOS-XE SDWAN routers. Validating data plane paths for a given prefix with vEdge is easier since the code was designed with this in mind. We can validate it straight from vManage.

Because there is only one WAN Edge router at BR1, there are only two possible transports to this prefix, one per TLOC atttached to BR1-WAN1. However, remember that two TLOCs does not necessarily mean two paths.

I’ve highlighted the important data plane tunnels here. Even though the routing table clearly shows only two next-hops, in reality there are four data plane tunnels established. This is what is vital to understand about TLOCs: TLOCs are an abstraction and not a physical attribute. The TLOC does have some physical attributes, but it is not the same as a true tunnel endpoint in the terms of traditional IPSec tunnels. Especially when dealing with cross-color data plane tunnels, the idea of TLOC-as-physical-next-hop breaks down quickly.

How Likely Am I To Run Into This?

Of course, the idea of cross-color tunnels only comes into play when dealing with different transport colors that provide connectivity across them. In a traditional setup with MPLS and Internet transports, as an example, this is unlikely to be possible. However, it was this cross-color tunneling which was providing a lot of misunderstanding and confusion and so I wanted to address it, and the concept of TLOC here. The main takeaway is to remember that a TLOC is not a physical next-hop in terms of data plane. It is a map to a specific WAN Edge on a particular transport color.

From the data plane perspective, what matters is whether the cross-color data plane is allowed or even possible. This is more like traditional IPSec where colors don’t even exist. If you can reach the tunnel endpoint you can build a tunnel. TLOCs provide (via the vSmart TLOC Exchange) the information needed to build those tunnels.

I hope this helped rather than hindered your understanding of TLOCs, please let me know if you have any feedback. Thanks!

Further Reading

Cisco CVD: Cisco SD-WAN Design/Deployment Guide

Cisco Live US 2019 Breakout: Delivering Cisco Next-Generation SD-WAN with Viptela

Carpe DMVPN: Deep Dive: vSmart Controller