On the suggestion of some of my Twitter followers, I’m going to publish some content around how Aviatrix moves packets within a cloud network. This will be fairly basic, but should give a good primer for anyone wondering what role Aviatrix can play in cloud networking.
First, here’s a simple topology that will help in the discussion.
This is a fairly basic deployment for a cloud network. In this setup, we have four VPCs, two each across two AWS regions. Each VPC has been subdivided into four subnets across two availability zones for high availability and redundancy.
Two of the four VPCs will be used primarily for some sort of workload, whether that be an application database, EC2 instances, some managed SaaS offering, basically any cloud-based computational work.
The other two VPCs will be used to provide cross-region transit. Strictly speaking, if there were really only two workload VPCs in the entire deployment, it would be unnecessary to deploy transit-based VPCs. We could simply do native AWS VPC peering to connect the VPCs across regions. In our example, we’ll only be working with a few VPCs, but in an actual deployment there could be tens, hundreds or thousands of VPCs and that would scale far beyond basic native peering and require more complex networking such as AWS Transit Gateway. In a later post we’ll explore why the cloud-native options may be undesirable even at low scale depending on the requirements of visibility and security; for now, let’s move on.
Cloud Native Networking
I’ve done some videos on my YouTube Channel around basic cloud native networking, but I’ll reiterate a little here to tie into the packet flow. Each VPC (or vNet in Azure, etc.) has a hidden router which handles local routing within that VPC and egress from it based on the route table. That hidden router cannot be configured or viewed, but its behavior can be influenced by updating the route table associated with subnets created inside the VPC. As we’ll see, this is a vital piece of all cloud networking.
Because that router is hidden and inaccessible for troubleshooting, visibility and configuration, the cloud native networking offer doesn’t provide more than basic networking, moving packets from Point A to B. The cloud was designed for developers, only basic networking was needed and was not a focus for CSPs.
For many reasons, the trend of moving critical workloads to the cloud began earlier than anticipated (COVID-19?) and this level of networking support is very challenging for teams who have, for years, had complete control and troubleshooting capabilities in a traditional data center.
Aviatrix to the Rescue
Aviatrix creates a data plane on top of the CSP’s native network, allowing much more granular control of the routing between VPCs, true packet-layer visibility of traffic crossing the data plane, and security of the workloads attached to that data plane. There’s far too much to cover in a single post, so for now we’ll just start at the beginning. Let’s deploy a basic hub and spoke model based on the diagram above and get two workloads talking to each other.
The most basic data plane element of the Aviatrix solution is the gateway. Without splitting too many hairs, the gateway is essentially a virtual router that can handle the route intelligence and forwarding necessities of cloud workloads. It is cloud native, lean, built to solve the networking challenges of the cloud itself. Because of this, the humble gateway can often be sized far under the requirements of traditional network vendor routers. There’s no extra code or functionality that’s useless for cloud deployments, and this lets the gateway focus on what is important: routing, forwarding, and telemetry of data.
Of course, the real hero is the Aviatrix controller. I haven’t included it in the diagram because the controller is not in the data plane, but it handles the deployment of gateways, it can orchestrate cloud native networking route tables, subnets and VPCs (as we will see), and it serves as the routing intelligence of the solution as well, distributing routes between gateways per policy. Gateways maintain a control tunnel to the controller for telemetry, route updates and health checking. If gateways fail, the Aviatrix controller can replace it or update route tables on the fly to heal traffic flows.
Pictured above is a standard deployment for gateways within a VPC. Note that these are spoke gateways deployed in an active/active high-availability fashion. Gateways can be deployed in single or HA mode, though the preferred method is in HA mode for obvious reasons. When deployed in HA mode, each gateway is deployed in a different availability zone. The gateways build a tunnel to each other to share traffic, and the controller actually programs the route tables of the VPC to load balance traffic between the gateways.
We deploy the Aviatrix gateways in a public subnet because each gateway uses a public IP to build the tunnels for communication with the controller, as well as inter-VPC IPsec data plane tunnel setup.
As of this writing, there is also a private-only option to deploy gateways using AWS PrivateLink instead of public IPs, but I’ll focus on the ‘standard’ deployment here. I want to emphasize that though the gateways have public AWS Elastic IP addresses, the gateway communication is kept within the AWS backbone and the gateways themselves are protected by multiple layers of security. The gateways are hardened via controller-orchestrated Security Groups as well as local hardening on the box to only allow required services from known peers.
Creating a Data Plane
Next, we attach the spoke gateways to the transit gateway. The spoke gateways reside with the cloud workloads, the transit gateway’s main function is to route traffic between spokes or to other destinations. This allows traffic scaling to occur where multiple spokes can communicate without needing to be peered together directly.
This tasks the gateways with creating a mesh of IPsec tunnels between them in what Aviatrix calls the ActiveMesh. This is an active/active traffic forwarding tunnel mesh between primary and HA gateways and is best visualized via the Aviatrix CoPilot topology tool. First, a diagram:
Now let’s see that visually via the CoPilot platform, which is Aviatrix’s operations-level troubleshooting and visualization offering.
I won’t double up on screenshots and explanations here, but per the topmost diagram, we do the same workflow in the other AWS region: Deploy spoke gateways, deploy transit gateways, and build that data plane between them.
The next step is to get the transit gateways peered so that the private workloads can communicate.
You may have noticed that between transit peers, we have the ability to do route filtering. This can be really useful if you prefer local cloud egress toward the Internet, as an example, and do not want AWS workloads to move across region/clouds to ingress/egress the cloud.
Let’s take a look at the result of transit peering.
Here is the CoPilot visualization of the resulting data plane.
At this point, the data plane should work. To recap the workflow up until now, the Aviatrix controller took care of the following:
- Deployed spoke gateways to workload VPC
- Built/applied security groups for spoke gateways for security hardening
- Orchestrated IPsec tunnel between primary and HA gateway
- Orchestrated native cloud route tables to point non-local routing to Aviatrix gateways by load balancing each RT to one of the two gateways
- Deployed transit gateways to transit VPC (and all the aforementioned tasks for the gateways as well)
- Attached the spoke gateways to the transit gateways via IPsec tunnel meshing for equal-cost load balancing and redundancy
- Orchestrated transit peering across regions to allow spoke-spoke communication
- Orchestrated all route sharing between gateways
One last setting to take care of. We need to turn on Connected Transit to allow spokes to communicate with reach other using the transit. The default setting assumes that spokes should be islands, we turn on the CT feature to allow the transit to forward packets between spokes instead of just spoke to ingress/egress.
Walking Packets Through the Data Plane
I have deployed test hosts to each private workload VPC and a bastion host to one of the public subnets so I can log into the private hosts for testing. The next section will focus on a ping from host to host across regions. I’ll dissect each hop and route decision to finish this post.
Hop 1
This traceroute is from the private host in Spoke 1 VPC to the private host in spoke 2 VPC in another region. I used the -I flag on the traceroute to ensure it used ICMP since that was allowed by the Security Group.
Let’s break this down hop by hop. How did the packet get to 10.10.66.94 and who is it?
First, we need to see where the packet is coming from on its way to the host in spoke 2. The screenshot above shows that the spoke 1 private host’s network interface is in the Private-1-us-west-2a subnet, so we need to look at the route table for that subnet to see what routing decisions are made at the first hop.
The route table says that the next hop for the destination will fall inside the 10.0.0.0/8 subnet and the next-hop is an elastic network interface. Let’s figure out what that ENI belongs to.
This makes sense. The ENI belongs to the Aviatrix primary gateway inside the spoke 1 VPC. The packet will go from the host in the private subnet to the spoke gateway. What happens next? We’ll look at the gateway route table now.
The destination IP is 10.20.2.42 which is the private host in spoke 2 VPC. Per the Aviatrix gateway route table, there are several tunnels available to reach that network. Remember that the gateways build an active mesh of IPsec tunnels and use ECMP to forward traffic. The first set is the transit gateway in region, which makes sense. The HA spoke gateway also advertises reachability to that network, but with a lower metric. If one of the tunnels or gateways in the mesh fails, there’s no need to panic as multiple paths exist.
Hop 2
From the traceroute, we know the next hop IP is 10.11.0.102. What gateway is that?
The packet was forwarded to the HA Aviatrix transit gateway in the transit VPC. This was just a result of traffic hashing, it could just as easily have ended up on the primary. Time to look at the transit gateway route table.
The reason we don’t need to look at the native VPC route table any more is because all traffic was encrypted in the IPsec tunnel and sent directly between gateways. The only thing the cloud native route tables in the transit VPCs are doing is providing reachability between the Aviatrix gateway elastic public IPs for the tunnel.
Here we see the tunnel points to the other region’s transit gateway for the next hop. This makes sense, these two transit gateways are peered across regions. Briefly, let’s inspect that peering.
Okay, now to see what 10.21.0.77 (the next hop in the traceroute) is.
Hop 3
The next gateway is the primary transit gateway in the other region. By now, this should be old hat. Let’s take a look at its route table.
This is not really surprising. The next hop is across the IPsec tunnels to the spoke 2 gateways. Consulting the traceroute, the next hop is 10.20.40.201. We can actually save a step (and could have before, but I wanted to illustrate the packet flow from soup to nuts) and consult the gateway route table. 10.20.40.201 is reachable over the tunnel to the primary spoke gateway in the spoke 2 VPC. We just need to take a look there and then the spoke 2 VPC native route table to finish up.
Hop 4
The spoke gateway shows the next hop to 10.20.32.0/20 is reachable via eth0 (the ENI of the gateway in the VPC). All that’s left is to see what subnet the Aviatrix gateway is deployed into and how the VPC route table looks.
The subnet of the gateway is Public-1-us-west-1a. We could look at the route table in the AWS console as before, but the Aviatrix controller can also show us.
Hop 5
The packet to 10.20.2.42 will be forwarded directly to the host by the VPC router. The return traffic will take the same path in reverse to reply to the ICMP packet.
Wrapping Up
This concludes a very basic look at an Aviatrix packet walk. There are a lot of use cases we didn’t even have time to talk about, much less demonstrate, and that will have to be the subject of a future post. Next time we’ll deploy firewalls into the Transit VPC and demonstrate how packet walks through the Aviatrix Transit Firenet works.
