Disclaimer: the content of this blog is solely based on my personal view/experience, and it’s not a company or someone else’s view. The content is intended for educational purpose only, and it’s not an official whitepaper or best practices document. Therefore, you must always refer to the official and latest AWS documentations, before considering anything discussed in this blog series, in any AWS environment.
The previous blog “AWS Cloud Networking Foundational Concepts” discussed the basic foundational concepts of VPC IP network and how it fits within the bigger picture of the AWS global infrastructure.
With the information covered in the previous blog, you could have provisioned an EC2 instance with an IPv4 address taken from the VPC CIDR range. So how does the IP allocation to the EC2 instance happened?
It’s the magic of DHCP, yes in each VPC private IPs to instances are allocated through DHCP aka DHCP options set, you can create as many additional DHCP options set as you want. However, you can only associate a VPC with one set of DHCP options set at a time. After you create a set of DHCP options, you must configure your VPC to use it.
For more info:
From networking point of view, we know that if a host wants to communicate with a destination IP that is outside its local IP network/subnet, it needs a default gateway/router to route the Layer 3 packet following basic routing concept depicted below.
So where we can configure this in AWS VPC? The Simple answer, its included as part of the DHCP and cannot be changed (unless you add a static route at the host level to point to a different IP). When a subnet is created, and technically allocated to an AZ (you may have multiple subnets per AZ) the first IP in the IP range, e.g. “.1” is assigned as the default gateway that’s maps to the subnet virtual router, to route packets from and to the subnet.
We all know, if there is a virtual router, there must be a route and this means we need a route table.
Does this indicate there is a routing table in a VPC? That’s right, in an AWS VPC you can have one or multiple route tables. In a VPC, there is “main route table” that automatically comes with the VPC, this route table is used to route traffic for all subnets that are not explicitly associated with any other route table. Also, you can create one or more “custom route table”.
So why someone might need more than a route table?
Before answering this question, let’s see how an instance can access the internet with a single route table. As we highlighted earlier, an instance will pick an IP from the VPC DHCP, which is within the VPC CIDR range (private IPv4). In order for an instance to access the internet, first we need to add an Internet Gateway as part of the VPC, and associate it with a route table.
“An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet. It therefore imposes no availability risks or bandwidth constraints on your network traffic.”
We know, to communicate over the public internet, an instance needs a public IP. To assign a public IP to an instance, we can either, specifically assign one at the instance launch, or by modify the subnet’s public IP address attribute (enabling auto-assign public IPv4 address).
The above, is applicable to any non-default VPC. While, the default VPC, already includes an internet gateway, and each default subnet is a public subnet. Each instance that you launch into a default subnet has a private IPv4 address and a public IPv4 address. In both cases, we will have a design similar to the one depicted below:
Note: from packet filtering point of view, a VPC Network Access Control List NACL ( at a subnet level) and Security Group at an instance level need to be configured to allow the desired traffic to pass through.
As you may noticed, the instance within a VPC uses its private IPv4 address, however, the assigned public IP address will come into effect when a packet reach the internet gateway that will translate it using the assigned public IPv4 address (NATing), this is something automatic and no need to do any configuration.
In other words, the Internet Gateway IGW in a VPC serves two key purposes: first to provide a target in the VPC route table(s) for Internet traffic, second to perform NAT for instances that have been assigned public IPv4 addresses. even though IGW supports IPv4 and IPv6, there is no NATing for IPv6. This means all IPv6 IPs are internet reachable. However this can be manipulated using the concept of egress only IGW if such communication model is required.
For more info about egress-only IGW refer to the below link
What if the design requires certain IPv4 instances to be able to connect to the internet, outbound direction only? E.g. an OS or application can download updates from the internet.
As we know, having an instance with a public IP, means this instance can be reached from the public internet. Also, we can not use egress-only IGW, as its for IPv6 only. By performing network address translation NAT, we could easily achieve the above requirement. However, if the instance subnet is public the instance will still be reachable from outside!
Earlier in this blog, we highlighted the ability to have more than one route table. So by having a second route table, and configure the associated subnet(s), to be private (no public IP assignment enabled), then we can take advantage of AWS NAT gateway that will be sitting between the two route tables and subnets.
In other words, route tables can be thought of ‘in a way’ like a source-based policy-based routing (PBR) concept. As illustrated below, Instance-A resides, in public subnet and the route table it uses, has a default route points to the Internet using the VPC Internet gateway. In contrast, Instance-B resides in a private subnet and its route table has a default route points to the NAT gateway. So here with this simple example, we were able to dictate which direction packets should go based on the subnet the instance associated with.
As it shown in the figure above, there is a NAT gateway per subnet/AZ because it is not a VPC-wide function like the Internet gateway, also we still need the Internet Gateway in the VPC. However, for the instances in private subnet that need to access Internet via the NAT gateway, the default route for where these instances, points to the NAT gateway.
One of the key design considerations with NAT gateway is to look at its limits, for example at the time of this blog writing, a single NAT gateway can handle maximum of 55K sessions to the same destination/IP, what if you need higher than this scale?
An alternative option, is to distribute the load from the sources inside the VPC (source instances) across different subnets/AZs) each with its own routing table, and provision a NAT Gateway per subnet. Alternatively, you may prefer to use your own virtual instance to perform NATing and might be combined with other functions like next generation firewall, IPS etc. whether you build on an EC2 instance or you obtain it from AWS marketplace (however, in the case you need to take care of the entire NAT service, in terms its availability, SW maintenance etc.), the logic of routing traffic to this instance should be the same.
Note: if this FW/NAT instance need to be placed in the same subnet as the internal EC2 instance for whatever reason, you will need to add route at the EC2 host/OS level to point to the FW/NAT IP as the next hop, as mentioned earlier in this blog, the subnet default gateway at the route table (.1 or any first IP in the IP range) cannot be changed.
This will lead us to the point where we need to take a step back, and think first, why an instance/host/application need to connect to the outside.
There could be multiple reasons and solutions, such as the scenarios covered above (an instance can connect to the internet as well as to be reachable from the internet, an instance needs to connect to the internet but shouldn’t be reachable by traffic initiated from the internet, or an instance need to access AWS public services in which reachability outside the VPC sourced from public IP is required NATed or without NAT)
What if the IP has to be static? > consider assigning AWS static public IP, also known as elastic IP address.
What if an application in a VPC only needs to connect to one or more AWS public services that are reachable through public IPs. For instance, connect to S3, DynamoDB, etc. without having any Internet Gateway and/or NAT gateway added to the VPC (this might be needed for a security compliance)
In such scenario, you can leverage a VPC endpoint, which enables you to privately connect your VPC to supported AWS services, including but not limited to: Amazon API Gateway, S3, AWS Glue, Amazon Kinesis, SNS, SQS DynamoDB etc.
The connectivity from a VPC private subnet will be established without requiring an internet gateway or a NAT device, as the VPC Endpoints allows resources in your VPC to use their private IPv4 addresses to communicate with resources outside of their VPC.
With this connectivity model, from AWS global infrastructure point of view, traffic between your VPC and other services does not leave the AWS network backbone. Which offers better performance.
Also, since, the traffic doesn’t traverse the public Internet, it means reducing the exposure to threat vectors such as brute force and distributed denial-of-service attacks. Also, it allows session initiation from the VPC side only, in other words, AWS cloud services cannot initiate a connection to resources inside a VPC through the endpoint, instead, only resources can initiate the connection and the VPC endpoint can return responses to traffic initiated from the VPC resources.
At the time of this blog writing, there are two types of VPC endpoints can be provisioned:
From its name, you can tell it is a gateway function, however this gateway function/endpoint, you specify as a target for a route in your VPC route table (which means you can add it, per route table), for traffic destined to a supported AWS service, at the time of this blog writing, Amazon S3 and DynamoDB are supported by the gateway endpoint.
From routing point of view, Gateway VPC endpoints use routes and prefix lists to route privately to the aforementioned AWS cloud services. As a result, Instances inside your VPC where the gateway endpoint is associated, will first resolve the service’s Domain Name System (DNS) to it’s public IP address, the route table will redirect traffic to these destination IPs/Prefixes toward the gateway VPC endpoint as the next hop, to access the intended service. This implies that you need to enable DNS resolution in your VPC.
Because each route//prefix-list has to be unique in a given route table, you can associate one VPC gateway endpoint per route table. If there is a need to apply sperate endpoint access polices for different instances, you may create different route tables and subnets each with its own VPC gateway endpoint.
As illustrated below, the VPC gateway endpoint is associated with VPC route table B, that has instance B’s subnet associated with it, as a result when instance B wants to access S3, traffic flow will go via the VPC gateway endpoint rather than using internet or NAT gateway. In contrast, route table A that has instance A subnet associated with, has no VPC endpoint associated with it, therefore, traffic from instance A to S3 will flow over the VPC IGW.
For more info:
Unlike gateway VPC endpoint, which relies on IP prefixes and routing table, interface endpoints are local IP addresses within the VPC, this IP technically associated with an elastic network interface ENI, in your VPC with a private IP address that serves as an entry point for traffic destined to a supported AWS service. However, it is zonal service, where you need to add it per availability zone.
When the interface endpoint is created, AWS automatically, creates regional and zonal DNS entries that resolve to local IP addresses within the VPC.
Note: When an interface endpoint is created, AWS generates endpoint-specific DNS hostnames that can be used to communicate with the service. For AWS services and AWS Marketplace Partner services, the private DNS option (enabled by default) associates a private hosted zone with your VPC. The hosted zone contains a record set for the default DNS name for the service (for example, ec2.us-east-1.amazonaws.com) that resolves to the private IP addresses of the endpoint network interfaces in your VPC. This enables you to make requests to the service using its default DNS hostname instead of the endpoint-specific DNS hostnames
These endpoints are also powered by AWS distributed networking architecture (Hyperplane) that was highlighted in the previous blog.
Note: interface endpoint powered by AWS PrivateLink can be used to provide ingress access to services in a consumer-provider manner. This topic will be covered in a future blog in more details
Unlike VPC gateway, access to VPC Interface endpoints can be extended to on-premises site over AWS DirectConnect or VPN.
Since the VPCE interface can be provisioned per AZ, in total you can go beyond 10G of bandwidth across multiple AZs (as aggregate, but each flow, will be limited to a single VPCE ENI capacity).
“By default, each interface endpoint can support a bandwidth of up to 10 Gbps per Availability Zone. and bursts of up to 40Gbps. If your application needs higher bursts or sustained throughput, contact AWS support.”
For more info about the Interface endpoint properties and design considerations, refer to the link below:
Note: Services might not be available in all Availability Zones through an interface endpoint, for more info refer to the link below: