Aviatrix Packet Walk: NAT, the Magic Dragon

Let’s be honest: NAT is one of the most complex and frustrating networking topics (outside of Multicast) that exist in modern networking. It’s enough to make you (almost) adopt IPv6. NAT is hard enough to do on-prem where everything falls under your control, but once the whole network layer is abstracted by a CSP, it’s impossible. I mean that literally, the native cloud network constructs don’t support NAT, so if you need it, you need a solution like Aviatrix to help. In the last post I covered a very basic 1:1 mapped NAT use case, which is already far and away beyond what a CSP can offer.

It’s time to tear off the band-aid and get a glimpse of how deep the NAT rabbit hole goes. Aviatrix supports damn near every possible scenario where NAT is a factor; I’m going to show two real-world scenarios where the native cloud would fail utterly at offering a solution, but where Aviatrix makes it simple (well, simpler). This is a packet walk so we’ll be going deep into the traffic flows, but lets start by simply setting up the scenarios.

By now this diagram should be getting familiar; we just keep adding to our cloud deployment. It’s time to add a second data center and deal with overlapping (and non-overlapping) traffic between that DC and the cloud.

Redundancy, Redundancy, Redundancy!

In this DC we have two routers, so let’s step through a Site 2 Cloud connection with redundancy. I covered most of the details in the last post so I won’t spend a lot of extra time here, I’ll simply focus on setting up the redundant connectivity.

Notice that this will be an Unmapped connection, so the NAT will not be a 1:1 setup. We also check the Enable HA checkbox which will add the extra fields for setting up the HA gateway and remote side second router. One more thing to see is that we have the ability to use a single remote router but build two tunnels (one from the primary and HA gateway respectively) if the remote end only has a single router with Enable Single IP HA.

We set up the primary and backup remote side routers and map the connection to the primary and HA gateways on the Aviatrix side, as well as specify the remote subnet to route traffic for. In a future post we’ll do BGP instead for routing over a Site 2 Cloud tunnel and show how easy it is to exchange routes that way, for now I wanted to focus on the NAT.

Just like before, after setting up the connection we can download the configuration. The difference in this case is that the config will include the primary AND backup config for the remote side.

A Tale of Two Traffic Flows

The first traffic flow to consider is from DC2 to a VPC workload that does not overlap in the WEST-2 Workload VPC. Let’s suppose an enterprise in a hybrid state, running some of the application resources in the cloud and others in the data center. A (simulated) client in DC2 needs to send data to a workload in the WEST-2 Workload VPC. There are multiple ways to accomplish this, but for purposes of demonstration I went with the idea that from a DC2 perspective, the entire cloud is represented as 10.110.0.0/16, an unused on-prem range. To facilitate cloud communication (and because dynamic routing will be in a future post) we need a static route from DC2 to the Site 2 Cloud tunnels. On the Aviatrix side, we need to set up the NAT features that are needed to allow this traffic.

Because the on-prem DC 2 IP addresses do not overlap with the target VPC, we don’t technically have to change the destination address, but remember that we decided to simplify the cloud-to-DC routing by representing the entire cloud with 10.110.0.0/16. That means that the DC 2 client will send its packet to an IP chosen from that range to represent the target workflow in the cloud. We’ve chosen 10.110.1.1 to stand in for the actual workload (although it could also be a load balancer front-ending a distributed workload). Our DNAT needs to be set up so that any packets arriving in the cloud via that Site 2 Cloud connection for 10.110.1.1 will have the destination IP changed to the real workload (load balancer, etc), in this case: 10.10.3.8.

Here’s the Aviatrix config for the destination NAT on that landing zone gateway pair. It had to be split into two screens because of all the options available to customize the DNAT criteria:

First, notice that we can sync this destination NAT config to both primary and HA gateway. The reason is because we are only changing the destination IP, not the source. Either gateway can make that swap without impacting traffic flow. For match criteria, the source IP has to fall within the 10.20.0.0/16 range, the DC2 data center.

Astute engineers might have already thought, “But 10.20.0.0/16 is also being used in the cloud, for the WEST-1 Workload VPC! How will the gateway know what to do?” The answer to that lies in the Connection/Interface setting. For DNAT, the rule will match based on where the packet is coming from. This means that the DNAT configured will only trigger based on a packet coming from the DC2 Site 2 Cloud tunnel. We also match on the destination IP of the packet to make sure we are doing a DNAT for the right destination.

The DNAT IP is the translated destination IP. We can see that for the WEST-2 workload DNAT (10.110.1.1/32) we are changing the destination to 10.10.3.8, the real IP. The other IP we will cover in a bit, when we talk about the second traffic flow.

NAT Order of Operations

Before we see the next step, let’s spend a little time talking about the NAT order of operations in an Aviatrix fabric to help give it context. When a gateway receives any packet, it will check for the requirement to do NAT if a configuration exists, in the following order:

1. Destination NAT Check: Run packet through DNAT rules to see if it matches and if so, change destination IP per that rule
2. Firenet/Firewall Redirect: If this inbound connection has a policy for Firenet redirection, use the load balancing tuple based on new destination IP and send to firewall
3. Routing: On return from Firenet (if redirected), use the new destination IP to do a route lookup and select the appropriate egress interface to send from
4. Source NAT Check: Based on the selected interface to send from, check against any SNAT rules to see if the source IP address of the packet should be changed and then forward. Note: In the case of SNAT, the Interface/Connection parameter is part of the check for outgoing traffic, unlike DNAT which is checked on where the packet came from.

We’re going to see this as soon as we start walking the packet, but first let’s look at how the SNAT is configured with the context of the order of operations:

The match criteria for the source and destination IP range seems very wide, but in context it should make perfect sense. This landing zone gateway has only two possible connections. It either sends packets to the on-prem data centers or into the Aviatrix fabric. If the source is coming from 10.20.0.0/16, the DC2 IP address range, destined to anything in the 10.0.0.0/8 range, and, equally important, if the outgoing connection for routing that packet is into the cloud, towards the Aviatrix transit gateway, we want to use the SNAT. The source, 10.20.0.0/16 already exists in the cloud, so we can’t allow that to remain.

We change the source IP of any packet from DC2 outgoing to Aviatrix transit to be the IP of the gateway itself, similar to how we would do with a firewall doing internet connectivity. The 10.13.0.0/16 space is routable within the Aviatrix fabric and will be routed back to the same gateway.

Let’s see the similar config on the landing zone HA spoke gateway. Notice we can’t sync the SNAT config to the spoke, and with good reason:

The SNAT IP is different for the HA gateway, and this is of course by design as no two devices can have the same (real) IP. This also ensures symmetry. When the return traffic comes back to be delivered to DC2, the traffic will return to the same gateway to have the DNAT reversed.

Let’s walk a packet through the flow to pull this all together.

Packet Walk: DC2 to Non-Overlapping VPC (WEST-2)

Let’s start with the simulated client in DC2 (10.20.255.254) and walk through the connection to the workload in WEST-2 using 10.10.3.8.

Since the traffic is one way (DC2 into the cloud), and since the CSR simulating our DC2 router is very bad at showing the packet path due to the NATs, we’ll leverage the Aviatrix Controller’s troubleshooting capabilities to show the packets. Using the controller, we can capture packets right out of the data plane for inspection, something much harder to do with native cloud networks.

In this case we just captured everything and used Wireshark to cut out the noise, but we have the ability to narrow down what we capture if this were a much busier environment.

Now to take a look at the capture.

Remember that 10.20.255.254 is our simulated client in DC2, and 10.110.1.1 is the NAT address for the real workload at 10.10.3.8 inside the cloud. The very next packet shows the DNAT and SNAT accomplished. As a refresher, the steps look like this:

1. Packet sent from DC2 across tunnel to Aviatrix landing zone spoke gateway. Source IP: 10.20.255.254. Destination IP: 10.110.1.1
2. Packet arrives at gateway. Gateway checks DNAT rule, finds source IP, destination IP, and incoming connection matches the rule. Gateway changes destination IP to 10.10.3.8. Source IP remains unchanged.
3. No Firenet, so skip that check.
4. Check the route table for the next hop of 10.10.3.8, along with the interface to forward on. It should be forwarded to the attached Aviatrix transit, which is important for the next check.
5. Packet is checked against SNAT rules now that routing decision is made. The source and destination IP match along with the outgoing interface/connection. SNAT takes place, source IP changed to the IP of the gateway, 10.13.32.149. At this point the packet matches the Wireshark: Source IP 10.13.32.149, Destination IP 10.10.3.8.

Let’s jump over to the Aviatrix transit and see this traffic flow continue.

The source IP and destination IP have not changed after forwarding to the Aviatrix transit. From here it’s just a matter of normal transit-spoke routing, which we’ve covered in a previous post.

We’ll check in again when the packet arrives at the WEST-2 Workload VPC and see what the spoke gateway there shows:

In this case, because of the ActiveMesh behavior, the ingress and egress flows ended up hashing to different gateways. The ingress flow (from DC2) was delivered to the HA gateway, and the response (based on what subnet the workload resides in, and the Aviatrix controller load balancing route tables) came via the primary gateway. This doesn’t cause any issues for the traffic itself.

Returning to the transit gateways, we can see the source and destination IP addresses flipped as expected for a response. Because the source IP is an IP the fabric already knows about, we didn’t need to do any specific advertisements to attract that traffic back to the landing zone gateway.

Finally, we see the traffic return to the landing zone gateway, have the source and destination NATs undone, and the reply returned to the originating DC2 IP address.

Here’s a diagram of the packet flow end-to-end that we just examined, numbered by hops:

NAT only takes place on the landing zone gateway (Hop 1 and Hop 10), the rest is just routing.

Packet Walk: DC2 to Overlapping VPC (WEST-1)

In the case on the second flow, DC2 will send traffic to the WEST-1 workload, which has an IP address overlap problem. Both DC2 and WEST-1 Workload VPC use 10.20.0.0/16, meaning that the behavior of the NAT will necessarily be different. We’ll use 10.110.1.2 to stand in for the WEST-1 workload IP from the DC2 perspective. Let’s look at how.

Above we can see that just as with the previous flow, the rule is based on the source IP (not pictured), destination IP, and the incoming connection. We are going to DNAT 10.110.1.2 to 10.200.2.42.

But why do that? Why not DNAT it to its real destination, 10.20.2.42, the workload in WEST-1 VPC?

Let’s recall the NAT order of operations in context of this flow, and the problem will become evident.

1. Packet sent from DC2 across tunnel to Aviatrix landing zone spoke gateway. Source IP: 10.20.255.254. Destination IP: 10.110.1.2
2. Packet arrives at gateway. Gateway checks DNAT rule, finds source IP, destination IP, and incoming connection matches the rule.
3. Instead of following the actual configured behavior, let’s change the destination IP to the real IP of the workload like we did for the previous flow. The destination IP would now be 10.20.2.42. Source IP remains unchanged.
4. No Firenet, so skip that check.
5. Check the route table for the next hop of 10.20.2.42, along with the interface to forward on… but wait! We have a source IP of 10.20.255.254 and a destination of 10.20.2.42 when we go to do the route lookup!

This packet is going nowhere fast because the DNAT took place before the route decision. What we need to do is make sure the DNAT address will not overlap with a local route table decision in which the source and destination belong to the same network.

This is why we instead DNAT the packet to a virtual address, 10.200.2.42. To support the routing of this packet to its real destination, we have to do some other work first on the Aviatrix spoke gateways in the the WEST-1 Workload VPC.

This is the DNAT configuration for the Aviatrix spoke gateways in the WEST-1 Workload VPC. Notice the source IP address in the rules match the two IPs for the landing zone gateways as that will be what is used in the cloud. The destination IP is the virtual IP we configured for DNAT on the landing zone gateways as shown above, and last, the incoming connection (as this is DNAT) is from the Aviatrix transit gateway.

The DNAT IP is the real IP of the workload at 10.20.2.42.

To attract traffic meant for 10.200.2.42 to these Aviatrix spoke gateways performing the DNAT operation we need to advertise that CIDR into the fabric. We can do that manually via the controller:

Now that I’ve explained how that part works, let’s jump back to the landing zone gateways and show the traffic flow again.

Once More, With Feeling

1. Packet sent from DC2 across tunnel to Aviatrix landing zone spoke gateway. Source IP: 10.20.255.254. Destination IP: 10.110.1.2
2. Packet arrives at gateway. Gateway checks DNAT rule, finds source IP, destination IP, and incoming connection matches the rule. Gateway changes destination IP to 10.200.2.42. Source IP remains unchanged.
3. No Firenet, so skip that check.
4. Check the route table for the next hop of 10.200.2.42, along with the interface to forward on. It will be forwarded to the attached Aviatrix transit, which is important for the next check.
5. Packet is checked against SNAT rules now that routing decision is made. The source and destination IP match along with the outgoing interface. SNAT takes place, source IP changed to the IP of the gateway, 10.13.32.149. At this point the packet looks like this: Source IP 10.13.32.149, Destination IP 10.200.2.42

Let’s do what we did with the first flow and use Wireshark to verify.

As expected, the packet coming from DC2 arrives at the landing zone gateway with a source of 10.20.255.254 and destination of 10.110.1.2. The Aviatrix gateway performs the DNAT first, swapping 10.110.1.2 for 10.200.2.42. The source IP is unchanged.

The route table is consulted (there is no Firenet) and the egress interface is identified because the WEST-1 VPC spoke gateway was set to advertise 10.200.0.0/16 as a connected network. The SNAT rules are consulted next, and as the traffic pattern matches (along with egress connection) the SNAT is performed, swapping 10.20.255.254 for 10.13.32.149. The traffic is forwarded to the Aviatrix transit gateway.

At the transit gateway, this is how the traffic looks. If we consult the routing table of the gateway, we should see where to send packets destined for 10.200.0.0/16:

Per the routing table, packets destined to 10.200.0.0/16 should be forwarded to the Spoke 2 (WEST-1) gateways. Let’s inspect the traffic as it gets delivered, it should look different than the previous flow with no overlap.

Stepping through the traffic flow as traffic is delivered to the destination:

1. The packet arrives at the gateway and the DNAT rules are consulted. As shown in an earlier section, the source IP, destination IP and incoming connection all match, so the DNAT is performed and 10.200.2.42 becomes 10.20.2.42. The source IP is unchanged.
2. The ping packet is delivered to the actual workload.
3. The workload responds to the ping with a reply coming from 10.20.2.42 and destined to 10.13.32.149 (the landing zone Aviatrix gateway IP)
4. Reversing the flow, the NAT session is undone and the source IP is swapped from the real workload IP of 10.20.2.42, to the virtual IP of 10.200.2.42 (the destination IP this gateway received before performing a DNAT)
5. The route table lookup is done and there is no SNAT required, the packet is forwarded to the Aviatrix transit gateway for the return. Source IP is 10.200.2.42, destination IP is 10.13.32.149.

The transit uses the source and destination to forward the packet back to the landing zone Aviatrix gateway, which takes care of the backing out of SNAT/DNAT as in the first flow:

And that’s it. Here’s a diagram similar to the first flow showing the traffic flow by hop:

On this diagram, NAT takes place on Hop 1, Hop 3, Hop 6 and Hop 8.

Wrapping It Up

This is just one of near-infinite NAT options that Aviatrix can enable. We could get extremely granular, creating source/destination combinations of IP, port, connections and do all sorts of multiple NATs along the way, as needed. One of the most common NAT use cases outside of IP overlap is ingress routing, where a customer may want to reach an application load balancer which is configured to create connections to backend workloads. We’ll look into some of those use cases in the future.