Skip to content
FR

Determine High-Performing and Scalable Network Architectures

Step 17 of 28

AWS Network Architecture: VPC, Direct Connect, Load Balancing, and Edge

Where this lesson fits

This lesson covers Domain 3: Design High-Performing Architectures, Task 3.4: Determine high-performing and/or scalable network architectures. Domain 3 represents 24% of the scored content of SAA-C03. AWS network architecture connects workload placement, routing and edge delivery. Choose topology, hybrid links, load balancing and edge delivery to meet latency, throughput and growth requirements.

Security controls for the same components (security groups, network ACLs, WAF, Shield) are covered in VPC security and private connectivity, resilience in Multi-AZ recovery and failover, and charges in network transfer, connectivity, and edge costs.

Learning objectives

After this lesson you will be able to:

  • Design a VPC with public, private, and isolated subnet tiers across Availability Zones, sized for growth.
  • Read route tables and choose between an internet gateway, NAT gateway, gateway endpoint, interface endpoint, peering, and Transit Gateway.
  • Select a connectivity option for hybrid workloads: Site-to-Site VPN, Direct Connect, or both.
  • Choose between Application, Network, and Gateway Load Balancers.
  • Distinguish Amazon CloudFront from AWS Global Accelerator and from Route 53 latency routing.
  • Explain enhanced networking, Elastic Fabric Adapter, and placement groups for network-bound workloads.

1. VPC topology: subnets, routing, and addressing

A VPC is a Regional virtual network with an IPv4 CIDR block between /16 and /28. A subnet lives in exactly one Availability Zone. AWS reserves five IP addresses in every subnet (network address, VPC router, DNS, future use, broadcast), so a /24 subnet offers 251 usable addresses.

1.1 Tiered design

Tier Route to the internet Typical contents
Public subnet Route 0.0.0.0/0 to an internet gateway Load balancers, NAT gateways, bastion alternatives
Private subnet Route 0.0.0.0/0 to a NAT gateway (outbound only) Application servers, containers
Isolated (data) subnet No default route Databases, caches

Spread each tier across at least two (preferably three) Availability Zones, and in the zonal NAT model, put one NAT gateway per AZ so that an AZ failure does not remove outbound access for the others and so traffic stays inside its AZ.

1.2 Planning address space for growth

Choose a CIDR big enough for several years and for the other networks you will connect. Overlapping VPC CIDRs cannot be peered, and ordinary Transit Gateway routes do not resolve overlapping address spaces, so plan an organization-wide address scheme (for example 10.0.0.0/8 split per account and Region). If a VPC runs out of addresses you can add secondary CIDR blocks, but you cannot resize or change the original block.

Plan for IPv6 when the stem mentions IPv4 exhaustion: a common Amazon-provided allocation is a /56, with /64 subnets in the classic design; supported IPAM/BYOIP allocation sizes vary. Public IPv6 does not imply inbound reachability without routes and security rules. IPAM also supports private ULA/GUA IPv6 space. An egress-only internet gateway provides outbound-only internet connectivity for public IPv6 addresses without NAT; it does not make private IPv6 space internet-routable.

1.3 Route tables

Each subnet is associated with one route table. The most specific route wins (longest prefix match). The local route for the VPC CIDR cannot be removed. When a stem says "instances in a private subnet cannot download patches", look for a missing route to a NAT gateway; when it says "public instance cannot be reached", look for a missing internet gateway route, a missing public IP, or a security group rule.

Alternative: regional NAT gateways manage multi-AZ expansion where available. Compare their supported routing and charges with the zonal design; see Chapter 2 for NAT64/DNS64 and Chapter 14 for cost.

2. Reaching AWS services privately: endpoints and PrivateLink

By default an instance in a private subnet reaching Amazon S3 goes through a NAT gateway and the public endpoint, paying NAT processing charges and adding a hop. VPC endpoints avoid that NAT/IGW path. Traffic to AWS public service endpoints can also stay on the AWS network while traversing an IGW; public addressing is not proof of traversal outside AWS. Endpoint policies, IAM and resource policies still determine authorization.

Endpoint type Services How it works Cost
Gateway endpoint Amazon S3 and DynamoDB only Adds a prefix-list route to the route table No hourly or data charge
Interface endpoint (AWS PrivateLink) AWS services including S3, DynamoDB, SQS, SNS, Kinesis, KMS, Secrets Manager, ECR, plus partner/customer services An elastic network interface with a private IP in your subnet Hourly per AZ plus per GB
Gateway Load Balancer endpoint Third-party appliances Routes traffic through inspection Hourly plus per GB

AWS PrivateLink also lets you expose your own service to other VPCs and accounts through a Network Load Balancer and an endpoint service. It provides one-way, private access to a single service, without needing non-overlapping CIDRs, which is why it is the answer when the stem says "SaaS provider must expose one service to hundreds of customer VPCs, some with overlapping CIDRs".

3. Connecting VPCs: peering versus Transit Gateway

Feature VPC peering AWS Transit Gateway
Topology One-to-one connection Hub-and-spoke through a Regional router
Transitive routing No Yes
Scaling n(n-1)/2 connections for full mesh One attachment per VPC, VPN, or Direct Connect gateway
Overlapping CIDRs Peering not allowed Attachments can exist, but overlapping routes are not propagated normally; renumbering, service access or explicit translation is required
Cross-Region Yes (inter-Region peering) Yes (inter-Region peering between transit gateways)
Bandwidth No central bottleneck; no per-attachment limit Up to 100 Gbps per AZ per direction for a VPC attachment; check quotas and traffic distribution
Extra features Simple, lowest latency, no hourly fee Route tables for segmentation, multicast, ECMP over VPN, centralized inspection
Cost Data transfer only (cheaper within an AZ) Hourly per attachment plus per GB processed

Consider peering for a few VPCs; Transit Gateway for a growing hub-and-spoke or hybrid topology. Peering is not transitive: if A peers with B and B peers with C, A cannot reach C. A frequent trap is a VPN or Direct Connect attached to one VPC being unreachable from a peered VPC ("edge-to-edge routing is not supported"); Transit Gateway solves it.

4. Hybrid connectivity: Site-to-Site VPN and Direct Connect

Option Path Bandwidth Setup time Notes
Site-to-Site VPN IPsec over the public internet Standard: up to 1.25 Gbps per tunnel; large: up to 5 Gbps on eligible TGW/Cloud WAN configurations; two tunnels per connection Minutes Encrypted by default; variable latency; ECMP over a Transit Gateway can aggregate multiple VPNs
Accelerated Site-to-Site VPN IPsec tunnels entering the AWS network at the nearest edge location through Global Accelerator Standard up to 1.25 Gbps per tunnel; large 5 Gbps tunnels do not support acceleration Minutes Requires a Transit Gateway attachment; more consistent latency
AWS Direct Connect Dedicated private fiber to a Direct Connect location Dedicated: 1, 10, 100 or 400 Gbps at supported locations; hosted speeds depend on partner/offer Weeks Consistent latency and throughput, lower data egress rates; not encrypted by default
Direct Connect + VPN IPsec tunnel over a public or transit virtual interface Limited by VPN Weeks Encrypted private path

Key Direct Connect facts:

  • Virtual interfaces: private VIF (VPC via virtual private gateway or Direct Connect gateway), public VIF (AWS public endpoints such as S3), and transit VIF (Transit Gateway through a Direct Connect gateway).
  • A Direct Connect gateway connects one connection to VPCs in multiple Regions (and accounts).
  • A single connection is a single point of failure. Maximum resiliency uses connections at two different Direct Connect locations (four connections total for the highest level), while a Site-to-Site VPN is the common low-cost backup.
  • MACsec provides point-to-point layer-2 encryption across the Direct Connect cross-connect on supported 10, 100 and 400 Gbps dedicated links. It is not end-to-end application encryption across every segment; use TLS/IPsec where the requirement demands that scope.
  • In September 2026 AWS announced flat-rate pricing for dedicated 10G and 100G connections, as an optional model with a fixed fee covering DTO within a selected geographic tier. A port pair includes a standby connection, but validate distinct devices/locations against the required failure domain; a same-location pair does not survive that location’s loss. For the exam, remember the principle: Direct Connect is the answer for predictable bandwidth, consistent latency, and high sustained transfer volumes.

5. Load balancing

Load balancer Layer Best for Key capabilities
Application Load Balancer (ALB) 7 (HTTP/HTTPS, gRPC, WebSocket) Web apps, microservices, containers, Lambda Path, host, header, and query routing; weighted target groups; authentication through OIDC/Cognito; WAF integration; redirects and fixed responses
Network Load Balancer (NLB) 4 (TCP, UDP, TLS) Extreme throughput, ultra-low latency, non-HTTP protocols, static IPs One static IP per AZ (optional Elastic IP for internet-facing IPv4 NLBs), client-IP preservation depends on target type/protocol and configuration, handles millions of requests per second, PrivateLink provider side
Gateway Load Balancer (GWLB) 3 (IP, GENEVE on port 6081) Inline firewalls, intrusion detection, deep packet inspection appliances Transparent bump-in-the-wire; scales third-party virtual appliances
Classic Load Balancer 4/7 legacy Old applications only Not the answer for new designs

Additional points:

  • Cross-zone load balancing is enabled by default on ALB (no data charge between AZs) and disabled by default on NLB and GWLB (data charges apply if enabled).
  • Sticky sessions (cookies on ALB) pin a client to a target but reduce even distribution; prefer storing session state in ElastiCache or DynamoDB.
  • ALB target groups can contain instances, IP addresses, or Lambda functions; weighted target groups support blue/green and canary releases.
  • A hostname alias to an ALB in Route 53 should use an alias record, which is free and supports the zone apex.
  • When the stem demands "fixed IP addresses that customers can allow-list" for an HTTP application, use NLB (Elastic IPs per AZ) or Global Accelerator in front of the ALB, since ALB IPs change.

6. The edge: CloudFront, Global Accelerator, and Route 53

6.1 Amazon CloudFront

CloudFront is a content delivery network with hundreds of edge locations. It caches static content close to users and also accelerates dynamic content by terminating connections at the edge and using the AWS backbone to reach the origin.

  • Origins: S3 buckets (protect with origin access control, OAC), ALBs, API Gateway, or custom HTTP servers; VPC origins let CloudFront reach resources in private subnets, and in 2026 support WebSockets.
  • Cache behaviors by path pattern with TTLs and cache keys; include only required headers/cookies in the cache key; use origin request policies separately for forwarding.
  • Signed URLs and signed cookies restrict private content; field-level encryption protects sensitive form fields; AWS WAF attaches to a distribution.
  • Origin groups fail over eligible GET, HEAD and cached OPTIONS requests, not POST/PUT writes; CloudFront Functions (lightweight, sub-millisecond) and Lambda@Edge (heavier logic) run code at the edge.
  • It supports HTTP, HTTPS, WebSocket, and HTTP/3, and viewer-side ACM certificates must be in us-east-1; an origin such as an ALB uses a certificate appropriate to its Region.

6.2 AWS Global Accelerator

Global Accelerator gives you two static anycast IP addresses that enter the AWS global network at the nearest edge location, then routes over the backbone to the closest healthy regional endpoint (ALB, NLB, EC2, or Elastic IP). It does not cache.

Use it for non-HTTP traffic (TCP and UDP, such as gaming, VoIP, and IoT), for static IPs that must be allow-listed, and for fast traffic shifts that do not wait for DNS TTL expiry. Health detection, endpoint readiness and client reconnection still determine application recovery; existing sessions are not transparently moved. It supports traffic dials and endpoint weights for blue/green shifts across Regions.

6.3 Which edge service?

Requirement Choose Reason
Cache images, video, or software downloads close to users CloudFront Edge caching
Speed up an uncacheable HTTP API globally CloudFront (dynamic acceleration) or Global Accelerator Both use the backbone; CloudFront adds caching and edge logic
UDP or non-HTTP protocol, global entry point Global Accelerator CloudFront is HTTP/HTTPS/WebSocket only
Static IPs for allow-listing and failover without client DNS-cache expiry Global Accelerator Anycast IPs, no DNS caching
Route users to the lowest-latency Region using DNS Route 53 latency-based routing DNS-level, subject to TTL caching
Custom domain serving private S3 objects CloudFront + OAC with an S3 REST origin A website endpoint cannot use OAC/OAI and is not this private-origin pattern

6.4 Network-bound instances

For network-heavy workloads, use instance types with Elastic Network Adapter (ENA) enhanced networking (up to 100+ Gbps on large instances). Elastic Fabric Adapter (EFA) adds OS-bypass, low-latency inter-node communication for HPC and distributed machine learning. A cluster placement group packs instances into one AZ for the lowest latency between them; a partition group isolates failure domains; a spread group puts a small number of critical instances on distinct hardware.

7. Worked example: a global game backend

Scenario. StarForge serves global UDP players through EC2/NLB backends in three Regions. Partners require fixed allow-listed IPs. A read-heavy HTTPS match-history API and 500 MB patches serve millions of users; the studio exchanges 8 TB of telemetry daily with AWS.

Step 1: game traffic. The protocol is UDP, so CloudFront (HTTP only) cannot carry it. Place AWS Global Accelerator in front of the three NLB endpoints. Players hit two static anycast IPs at the nearest edge location and ride the AWS backbone to the closest healthy Region. Test health detection and player reconnection against the required recovery time. The two static addresses satisfy the allow-listing requirement.

Step 2: patches. Store patches in S3 and distribute with CloudFront using OAC. Cached downloads come from the edge, cutting latency and offloading S3. Use long TTLs and versioned file names.

Step 3: match history API. Use CloudFront in front of an ALB (or API Gateway) with short TTLs for cacheable read endpoints and TLS termination at the edge.

Step 4: telemetry transfer. 8 TB per day (about 0.75 Gbps sustained) calls for consistent throughput: a Direct Connect connection (1 Gbps would be too tight, so a 10 Gbps dedicated or hosted connection) into a Direct Connect gateway associated with one Transit Gateway per Region through a transit VIF, with a Site-to-Site VPN as backup. Attach each VPC to its local Regional Transit Gateway. A single Transit Gateway cannot directly attach VPCs from other Regions; configure supported Direct Connect gateway associations and routing, with inter-Region TGW peering where the VPC-to-VPC path requires it.

Step 5: intra-VPC private access. Add a gateway endpoint for S3 in each VPC so internal jobs do not traverse NAT gateways.

Why other answers are worse. Route 53 latency routing to the Regional NLBs would meet the "closest Region" idea but depends on DNS TTLs for failover and cannot offer fixed IPs for all three Regions. CloudFront for UDP is unsupported. A Site-to-Site VPN alone cannot guarantee consistent latency at 0.75 Gbps sustained over the internet. VPC peering of the three Regions plus the data center does not give transitive routing.

8. Comparison: choosing the network service

Need Best answer Typical trap
Private access to S3 from private subnets, no cost Gateway endpoint NAT gateway (works but costs)
Private access to SQS or KMS Interface endpoint Gateway endpoint (only S3 and DynamoDB)
Many VPCs and on-premises, transitive Transit Gateway Peering mesh
Expose one service to many customer VPCs PrivateLink (NLB endpoint service) Peering (CIDR overlap and broader routed scope requiring filters)
Fast, consistent hybrid link Direct Connect (+ VPN backup) VPN only
Quick, encrypted, temporary hybrid link Site-to-Site VPN Direct Connect (weeks to provision)
Path-based HTTP routing ALB NLB
Static IP, TCP/UDP, millions of rps NLB ALB
Inline security appliances GWLB ALB with security groups
Global caching CloudFront Global Accelerator
Global UDP, static IPs, fast failover Global Accelerator Route 53 alone

9. Common pitfalls

  • Assuming VPC peering is transitive or that a peered VPC can use another VPC's VPN or internet gateway.
  • Using one zonal NAT gateway for all AZs, creating a cross-AZ dependency and charges.
  • Selecting an interface endpoint for S3 when a free gateway endpoint satisfies the stem (unless on-premises access over Direct Connect is needed, which requires an interface endpoint).
  • Choosing ALB when the stem requires static IP addresses or non-HTTP protocols.
  • Recommending CloudFront for UDP or for gaming protocols.
  • Treating Direct Connect as encrypted, or as quick to provision.
  • Overlapping CIDR blocks between VPCs that must be peered or connected through Transit Gateway.
  • Putting unnecessary headers/cookies in the CloudFront cache key, fragmenting cache entries. An origin request policy can forward additional values without adding them to the key; include authorization-relevant dimensions or disable caching where required.
  • Relying on Route 53 TTL-based failover where "seconds" of failover are required.
  • Leaving cross-zone load balancing off on an NLB with uneven targets per AZ.

10. Exam tips

  • "Private, free access to S3 or DynamoDB" → gateway endpoint.
  • "Private access to other AWS services or SaaS" → interface endpoint / PrivateLink.
  • "Dozens of VPCs and on-premises" → Transit Gateway.
  • "Consistent latency, large steady transfers to on-premises" → Direct Connect; "backup" → VPN.
  • "Layer 7 path routing, containers, Lambda targets" → ALB.
  • "Static IPs, TCP/UDP, extreme performance" → NLB.
  • "Inline third-party firewall appliance" → GWLB.
  • "Global static IPs, non-HTTP, fast regional failover" → Global Accelerator.
  • "Reduce latency for global static content" → CloudFront.
  • "Lowest latency between HPC nodes" → cluster placement group with EFA.

11. Mini case study (with solution)

Scenario. MedConnect has 12 team VPCs plus one shared-services VPC in eu-central-1 and two hospitals. A partial peering mesh lacks transitive access; hospital VPNs into separate VPCs deliver variable 200–800 Mbps throughput. A new mobile app serves patients across Europe and North America through a REST API behind an ALB; patients upload 20 MB documents. The compliance team requires that S3 data requests succeed only through approved VPC endpoints, including requests with otherwise valid credentials. Patients must use the authenticated application rather than a direct S3 upload URL.

Questions. Redesign the network for scalability, performance, and the privacy requirement.

Solution.

  1. Inter-VPC: replace the peering mesh with a Transit Gateway. Attach all 12 VPCs and the shared services VPC, using separate route tables to segment production from development while allowing shared services access. Ensure the CIDRs do not overlap.
  2. Hospitals: order Direct Connect connections (two locations for resilience) and connect them through transit virtual interfaces and a Direct Connect gateway to the Transit Gateway; keep the existing Site-to-Site VPNs, now attached to the Transit Gateway, as backup. Use IPsec over Direct Connect or end-to-end TLS for required end-to-end encryption. MACsec may additionally protect an eligible cross-connect, but has a narrower encryption scope.
  3. Patient API: put CloudFront in front of a private ALB using a VPC origin. Uploads and downloads pass through the authenticated application; its IAM role accesses S3 through an approved gateway endpoint. Forward required authorization headers and disable shared caching for private patient responses. Size upload timeouts, body handling and application capacity for 20 MB documents.
  4. Endpoint-restricted S3 bucket: keep Block Public Access enabled, grant only the required application IAM permissions, and use a bucket policy denying S3 data access unless aws:SourceVpce matches an approved endpoint. Configure endpoint policies too. Direct browser presigned S3/accelerated uploads and a direct CloudFront OAC-to-S3 origin do not traverse this VPC endpoint, so they do not satisfy this stricter rule. An OAC-only design is valid for the separate requirement “bucket non-public, distribution allowed.”

Why not alternatives. Adding more peering connections increases the mesh to 78 links and still lacks transitive routing. Increasing the VPN count without a Transit Gateway cannot aggregate bandwidth to many VPCs. Global Accelerator does not enforce S3 endpoint restrictions or application authorization; the selected CloudFront design adds HTTP controls and a private origin. Making the bucket public, or merely disabling public ACLs without endpoint restrictions, does not enforce the stated endpoint-only data path. Validate both authorized endpoint access and denial outside it.

12. Glossary

  • Internet gateway (IGW): VPC component enabling internet routing for public subnets.
  • NAT gateway: managed outbound-only translation for private subnets.
  • Gateway endpoint: route-table-based private access to S3 and DynamoDB.
  • Interface endpoint: ENI-based PrivateLink access to a service.
  • Transit Gateway: Regional hub router connecting VPCs and on-premises networks.
  • Direct Connect gateway: construct connecting Direct Connect to VPCs across Regions.
  • Virtual interface (VIF): logical Direct Connect connection of type private, public, or transit.
  • ALB / NLB / GWLB: layer 7, layer 4, and layer 3 load balancers.
  • Anycast: one IP address announced from many locations.
  • OAC: CloudFront authentication to an S3 REST origin; the bucket policy must grant the intended distribution. It is not a VPC endpoint.
  • EFA: Elastic Fabric Adapter for low-latency node-to-node communication.
  • Placement group: cluster, partition, or spread placement strategy for instances.

13. Summary checklist

  • I can design public, private, and isolated subnet tiers across AZs with room to grow.
  • I can choose between gateway endpoints, interface endpoints, and PrivateLink.
  • I know why peering is non-transitive and when Transit Gateway is required.
  • I can choose among VPN, Direct Connect, and both, including resilience and encryption.
  • I can choose ALB, NLB, or GWLB from layer, IP, and protocol requirements.
  • I can distinguish CloudFront, Global Accelerator, and Route 53 latency routing.
  • I know ENA, EFA, and placement groups for network-bound workloads.

Sources