Throughout this series I will leverage screenshots and other source material that Cisco provides for the solution, including product documentation and Cisco Live sessions as well as material from the dCloud demo we are using for this series. Whenever I use non-original resources, I will provide a link to those resources at the end.
Note: The platforms discussed are current as of this publish date. Cisco will undoubtedly expand the offering to other routing platforms.
Note: Whatever opinions I express in terms of the platforms and their sizing/targeted deployments are generalized and are my opinions alone. Your Cisco Account Team/Partner knows you better than I do.
Note: Sorry about the lateness of this post. I just moved to a new house and have been busting my hump to get moved in (not done yet). Of course it’s also the holiday season, which is always busy anyway. For the two of you who will read this, my apologies. 🙂
The Data Plane and You
Now that we’ve covered the Cisco SD-WAN infrastructure, it’s time to tackle the most important and versatile element of the solution: The WAN Edge routers. We’ll start by covering what platforms are currently available, then recap some concepts that were introduced earlier in the series as pertains to how the data plane fabric is established. We’ll then briefly discuss policy which affects only the WAN Edges instead of the entire fabric. Most of the information surrounding high availability, firewalls and NAT will need their own posts, so I won’t touch on them here in the interests of brevity. I will, however, plan to create posts on those topics in the future.
WAN Edge Breakout
This isn’t an attempt to sell these platforms, so I’ll focus on the parts which are interesting as pertain to SD-WAN, specifically the supported handoff types and the throughput if documented, as well as the general sizing suggestion for where the platform would be deployed (though of course all enterprises are different).
Viptela Platforms
vEdge 100(B/M/WM)
The vEdge 100 platform is the small branch/store offering. Three models exist, offering a variety of Ethernet and LTE transport connectivity depending on the model. The WM model also includes onboard WLAN capability. The 10/100/1000 Mbps throughput provides some versatility with smaller circuits and backwards compatibility.
vEdge 1000
With a low profile and flexible SFP support, as well as 1Gbps speeds, this is the mid-level offering for the branch. If unsure about which platform to use, this is probably the safest bet in the vEdge line.
vEdge 2000
The vEdge 2000 is the clear choice for headquarters/large office deployments. Though it has less fixed ports, it supports a much higher 10GBps throughput with the same low profile as well as 10G optics.
vEdge 5000
The vEdge 5000 is the Cadillac router of the line, supporting 20GBps throughput and 4 network interface modules. It’s targeted towards the largest enterprises and data centers. It otherwise has feature parity with the vEdge 2000.
vEdge Cloud
The vEdge Cloud was intended to be run either in a cloud environment such as AWS/Azure or as an on-prem solution. Depending on the resources available, the vEdge Cloud features similar performance to the vEdge 100/1000.
Cisco Platforms
ASR1000
Right now, the SD-WAN solution is supported on the following ASR1K platforms:
- ASR1001-X / ASR1001-HX
- ASR1002-X / ASR1002-HX
The ASR supports 1G/10G (with license upgrade) SFPs. The SD-WAN throughput is dependent upon licensing. Of the Cisco platforms that currently support SD-WAN, the ASR platforms would make the most sense in large enterprise/DC deployments.
ISR4000
Right now, the SD-WAN solution is supported on the following ISR4K platforms:
- ISR 4221
- ISR 4321
- ISR 4331
- ISR 4351
Obviously this is a huge range in the ISR line. The takeaways from this list involve understanding there are a lot of SFP options available for handoffs, that the throughput is going to be heavily based on licensing and which ISR platform is used, and that generally, the ISR4K is targeted to branches of varying sizes. Notice that the largest ISR4K platforms are not yet supported, so for the largest deployments it may be necessary to go with an ASR1K for now.
ISR1000
Right now, the SD-WAN solution is supported on the following ISR1K platforms:
- 1111-8P / 1111-8P LTE
- 1117-4P / 1117-4P LTE
The ISR1K is a small branch/store offering. As new code is released, the ISR1K platform support for SD-WAN will increase. The platform has both fixed GigabitEthernet ports as well as SFP support for fiber, and some models have LTE capability. The major takeaway from this platform is that it supports most SD-WAN functionality, but for SD-WAN security it needs to be upgraded to 8GB of RAM.
ENCS5000
For those who want to virtualize the branch, the ENCS platform with ISRv is a possibility. The details and restrictions of ENCS/NFVIS is beyond the scope of this deep dive, but SD-WAN is supported on this platform using the ISRv virtual router.
CSR1000v
The Cloud Services Router should be familiar to everyone by now. SD-WAN support for the CSR1000v is upcoming and will be the platform of choice for cloud deployments in AWS/Azure for those looking to extend the SD-WAN fabric into the cloud.
Let’s Build a Fabric
No deep dive of the data plane forwarding devices in the SD-WAN fabric could be complete without focusing on how the fabric is formed and packets moved. Let’s step through the components of the fabric and discuss how data gets forwarded, starting with the underlay transport.
Remember from the previous post on vSmart that TLOCs are Transport Locators, a tuple of multiple data points associated with a WAN Edge (see the picture). From a control plane perspective, TLOCs perform a similar function as BGP next-hop addresses for purposes of routing. From the data forwarding perspective, and based on the topology policy applied, TLOCs can be thought of as tunnel/fabric endpoints. The fabric of overlay tunnels is basically built in this manner:
- WAN Edges are onboarded to the fabric. See this post for more information about this process. Important to note is that the vBond Orchestrator informs the WAN Edge if it is behind a NAT and needs to do NAT traversal in order to set up the fabric tunnels.
- After being onboarded, WAN Edges receive topology and control plane policies from the vSmart, and any local policies from the vManage, as discussed in this post.
- WAN Edges advertise local TLOC routes through OMP to the vSmarts.
- vSmarts, based on the control policy, share TLOC information with other WAN Edges. This includes the information needed to build an ipsec tunnel.
- TLOCs also contain various attributes, encryption keys being one of the most important . Since the vSmart knows about the encryption keys of all WAN edges in the fabric, the vSmart is able to relay these to the WAN Edges so they can build their data plane. This allows the Cisco SD-WAN solution to have greater scale as you don’t need to negotiate IKE phase 1 with every device in the fabric.
- BFD probes sent within the ipsec tunnels measure path health statistics for use in application-aware routing policies, if configured. If the BFD probes fail, the TLOC will be seen as down.
OMP Path Selection: Where Do the Packets Go?
After a WAN Edge is onboarded, centralized and local policies deployed, and the fabric created, where will the router forward packets? Of course the short answer is, “wherever we want”, but let’s see how administrative distance and path preference determines routing decisions.
From the Unicast Overlay Routing Guide:
- Check whether the OMP route is valid. If not, ignore it. An OMP route is invalid if the advertised TLOC is unreachable.
- If the OMP route is valid and if it has been learned from the same SD-WAN device, select the OMP route with the lower administrative distance.
- If the administrative distances are equal, select the OMP route with the higher OMP route preference value.
- On WAN Edge routers only, if the OMP route preference values are equal, select the OMP route with the higher TLOC preference value.
- If the TLOC preference values are equal, compare the origin type, and select one in the following order (select the first match):
Connected
Static
EBGP
OSFP intra-area
OSPF inter-area
OSPF external
IBGP
Unknown - If the origin type is the same, select the OMP route that has the lower origin metric.
- On WAN Edge routers only, if the origin types are the same, select the OMP route the higher router ID.
- If the router IDs are equal, a WAN Edge router selects the OMP route with the higher private IP address. If a vSmart controller receives the same prefix from two different sites and if all attributes are equal, the vSmart controller chooses both of them.
Here are some examples of choosing the best route:
- A vSmart controller receives a OMP route to 10.10.10.0/24 via OMP from a vEdge router with an origin code of OSPF, and it also receives the same route from another vSmart controller, also with an origin code of OSPF. If all other things are equal, the best-path algorithm chooses the route that came from the WAN Edge router.
- A vSmart controller learns the same OMP route, 10.10.10.0/24, from two WAN Edge routers in the same site. If all other parameters are the same, both routes are chosen and advertised to other OMP peers. By default, up to four equal-cost routes are selected and advertised.
A WAN Edge router installs an OMP route in its forwarding table (FIB) only if the TLOC to which it points is active. This is very similar to BGP where you won’t install BGP learned route into your RIB if you can’t recurse to the next hop address. For a TLOC to be active, an active BFD session must be associated with that TLOC. BFD sessions are established by each WAN Edge router, which creates a separate BFD session with each of the remote TLOCs. If a BFD session becomes inactive, the vSmart controller removes from the forwarding table all the OMP routes that point to that TLOC.
Localized Policy
Unlike centralized policy, which is deployed from the vSmarts to control routing and topology information, a localized policy is deployed directly from the vManage to WAN Edges. Localized policy is generally used for QoS and to effect local routing decisions without having to apply the same across the fabric.
We’ll dig further into localized policy in another post, but I want to call out how the localized policy differs from the centralized policy here. Because the localized policy only affects device templates that ‘opt-in’ via attaching the template, there is far greater granularity in how it is applied. As an example, if a certain carrier uses a particular QoS policy which differs from other carriers, you may wish to deploy a localized QoS policy to ensure proper marking and queuing is available through that provider’s network.
Next time, we’ll start digging into Templates, and learn how the different template types combine to build a scalable, modular framework for an SD-WAN deployment.
Links:
IOS-XE SD-WAN Install/Upgrade Guide
