LISA2017 Big Three Cloud Networking
4 likes•568 views
This document provides an overview of networking concepts in the "Big Three" cloud providers: AWS, Azure, and GCP. It discusses the physical and logical organization of resources, including regions, availability zones, and accounts/tenants. It also covers network substrates like VPCs, VNets, and VPC Networks, addressing, and properties of network interfaces and instances. The document aims to compare approaches across providers and provide design exercises to better understand implementation differences.
LISA2017 Big Three Cloud Networking
- 1. October 29–November 3, 2017 | San Francisco, CA www.usenix.org/lisa17 #lisa17 The Ins-and-Outs of Networking in the Big Three Clouds Chris "mac" McEniry
- 3. Topics • Network Substrates • Routing, Routing, Routing • Access Control 3
- 4. • Focus on the Big Three Cloud Service Providers (B3CSP) • Slides are indicated with what Cloud Provider it's talking about • Amazon Web Services • Azure • Google Cloud Platform 4
- 5. What about Cloud X? This will not be covered... 5
- 6. What's the best cloud? This will be covered... (5 seconds) 6
- 7. "It Depends" That's always the answer to questions without requirements... 7
- 8. Not Covered: "Legacy" Models • Focus is on current practices. Some legacy practices are not even available to new accounts. • Two Main Areas • Organization: Some providers have a first pass on account or resource organization. Not Covered. • Networking: Each provider has a first pass on their network offerings. Not Covered: • EC2 Classic • Virtual Network Classic • Legacy Networking 8
- 9. Not Covered: IPv6 Addresses • Changes assumptions about Internal/External IP allocation • Not the majority of use cases (yet) • Support • AWS: Yes • Azure: ? • GCP: No 9
- 10. Beware of Limits/Quotas • http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/ VPC_Appendix_Limits.html • https://docs.microsoft.com/en-us/azure/azure-subscription-service- limits#networking-limits • https://cloud.google.com/compute/docs/vpc/#quotas_and_limits • https://cloud.google.com/router/quotas 10
- 11. Design Exercises • Pen and paper exercises • Goals • Better understanding of concepts • Compare/Contrast implementations in the B3CP • Best to attempt same goal with all three providers, but one is sufficient for this tutorial. • Some scattered throughout. More at the end • Write down assumptions you make when designing 11
- 13. • 9/20 VPC Resizing: Adding secondary CIDR • 11/1: Direct Connect Gateway, 3 more CloudFront POPs, Direct Connect Pricing Update • (11/27: re:Invent) • Early 2019: Bahrain Region 13
- 14. • 9/21 Availability Zones • 9/26 (Ignite) VNet Service Endpoints. DDoS Protection for VNets. NSG using Application Security Groups, Service Tags, more rule combinations. 1 Gbps VPN connection. Monitoring ExpressRoute, S2S VPN. ExpressRoute Public access for MS Services, Route Filters. P2S macOS support. P2S AD Authentication. More ExpressRoute Partners. • 9/27 VNet Integration for Azure Storage and Azure SQL. Storage Firewalls. SQL Endpoints. • 10/31 Triple capacity in China 14
- 15. • 9/19 Sao Paulo Region • (10/3 Custom IAM Roles) • 10/7 Multiple (<=8) NICs GA. • 10/31 Mumbai Region. Dedicate Interconnect GA, more POPs. • 11/2 Faster SDN 15
- 16. Organizing 16
- 17. Organizing • Structure of CSP physical and logical/administrative build outs has impacts on network capabilities and topology • Examples • What do you need to account for to build resiliency into the infrastructure? • How do you maintain separation of administrative domains (e.g. account) while allowing traffic to traverse where needed? • Does it make sense to do large shared networks, or smaller finely tuned networks? 17
- 18. Physical Organization • All B3CSP have Regions • Geographically separate facilities hosting resources • All B3CSP have some sub-Region container (separate power, network hardware, etc) • Availability Zones • Fault Domains • Zones 18
- 19. Availability Zones, Zones • Separate Data Centers that are close to each other • AWS: "less than 2 ms latency between each other" • Directly exposed to customer • You have to choose where to put resources 19
- 20. Fault Domains, Paired Regions • Fault Domain is similar to AZ/Z but is not directly manipulated by customer • Specify intent by putting resources into Availability Sets • "I want my 5 web servers to not fail all at the same time" • Availability Set distributed across Fault Domains* • Paired Regions: Regions in the same geographic region but are kept separate (> 300 miles, not operated on at the same time, etc) • East US<->West US, UK West<->UK South 20
- 21. Locations 21 Regions AWS 16 44 Availability Zones Azure 26 (not counting Gov) 60 Fault Domains (not counting Gov) GCP 12 36 Zones
- 22. 22
- 23. 23
- 24. 24
- 25. Logical/Administrative Organization • Each CSP has a way of administering users and resource permissions • Comparisons: • Where users are • Where resources are • Where permissions are set • Fundamentally asking "Who can spin up a VM in this network? Who can stop a VM that is running in this network?" 25
- 26. (Billing) Accounts • Single unified item • Hard boundary which contains all resources associated with it - can't share • Can be in an organization, but only resources are not shared across - only policies • Can build IAM accounts inside of this (or roles which connect from other accounts) • All permissions are primarily set for actions (not targets) on the account* • Must change account/role into another account to operate on its resources 26
- 27. Account / Tenant • Single unified item - Microsoft or Organizational Account • Hard boundary which holds all resources associated with it • Subdivided into Subscriptions, and then Resource Groups • Access Control can be place on Subscriptions and/or Resource Groups • Resources bound to one Resource Group/Subscription at a time • Can move (with some restrictions) resources across Subscriptions/RGs 27
- 28. Cloud Resource Hierarchy • Organization • Root of hierarchy • Projects • Core organizational component • Associated with Organizations (or stand alone) • Can have own permissions (and inherit) • Resources • Associated with Projects • Can have own permissions (and inherit) • In some cases, can be shared with other projects 28
- 29. Tags • Instances can be tagged • Tags are used in selection items • Firewall • Routing 29
- 31. Network Substrate • Virtual version of the traditional physical networks • Handles the basic packet forwarding • Organized into Subnets • Supports resilience - spread over different resources • Supports administrative separation - grouping of similar services 31
- 32. Not like a physical network • All packet forwarding based on some sort of mapping or hidden networking layer • Unicast only: No support for Broadcast or Multicast • No transit networking through the substrate 32
- 33. Virtual Private Cloud (VPC) • A single CIDR allocated to a Region in one Account • Subnets are smaller contained CIDRs assigned to an Availability Zone • Subnet CIDRs can't overlap • VPC CIDR can overlap regardless of Region/Account (as long as you don't want to connect them together) • Default VPC per Region (172.31.0.0/16) with /20 subnets in each AZ • Used for resilience and administrative separation 33
- 34. VPC Addressing • Can be RFC1918 or other IP space • AWS will not advertise space out to the Internet • Each object is a contiguous CIDR • Allocated at time of object (VPC, Subnet) creation • Can't change without destroying old/creating new 34
- 35. VirtualNet (VNet) • Single CIDR allocated to one Region in one Subscription • Subnets are smaller contained CIDRs that span Fault Domains for that Region • Subnet CIDRs can't overlap • VNET CIDRs can overlap regardless of Region/Subscription (as long as you don't want to connect them together) • Used for administrative separation 35
- 36. VNet Addressing • Can be RFC1918 or other IP space • Azure will not advertise space out to the Internet • Single CIDR at time of creation for VNet • Subnets can change CIDR if it's not in use at all • Must support a couple of special subnets • First contiguous CIDR of VNet : Used for Load Balancers • "Gateway Subnet" : Used for putting gateway devices (i.e. VPN) 36
- 37. Virtual Private Cloud Network (VPC Network) • Global private communication space allocated to a Project • Can be shared with other Projects • Subnets are single CIDRs allocated to a Region, and can span Zones • Two modes of allocation (one way switch from Auto -> Custom) • Auto Mode: Allocate a subnet to each region. Can manually add your own. • Custom Mode: Manually build subnets (recommended) • Each subnet has primary CIDR • (In Custom Mode) Can allocate secondary CIDR - typically for container networks 37
- 38. VPC Network Addressing • RFC1918 space only • Auto Mode: Predefined /20 to each region. E.g. • 10.128/20 -> us-central1 • 10.132/20 -> europe-west1 • Instances have primary IP and can have alias IPs • Alias IPs can come from primary CIDR or secondary CIDR • Can use CIDR (not just host) for Alias IPs on secondary • Secondary CIDR does not reserve network/gateway IPs 38
- 39. AWS Azure GCP Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network IP Addressing RFC1918 or Other Carving up CIDR of VPC RFC1918 or Other Carving up CIDR of VNet RFC1918 only Accumulation of Subnet CIDRs Locality One Region One Region Global Subnet Locality One AZ Region Selection of Zones in a Region CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR Account Resource Sharing No Users use multiple Subs. Resources bound to one VNet inside one Sub at a time Across Projects - Yes Across Organizations - No
- 40. Resiliency • Build out a Network Substrate • Capable of surviving a failure of one subregion area (AZ/FD/Z) • Supports a 3 Tier application (web/app/db) with clear delineation 40
- 42. Instance Properties • IP Forwarding / Source-Destination Checking • Checks whether a packet headed to the instance matches the IP(s) of the instance, or whether a packet leaving the instance matches the IP(s) of the instance • Instance NIC Properties • Number of IPs per NIC • Number of NICs per VM • Locations of NICs 42
- 43. AWS Azure GCP Forwarding Property Source/Destination Check IP Forwarding (enableIPForwarding) IP Forwarding (can-ip-forward) Property Default On Off Off NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface IPs per NIC 6-50 50 Unspecified (Alias IP not supported with multiple NICs) NICs per Instance 1-15 2-8 1-8 NIC Locations Same VPC Same VNet Each must be on separate VPC Networks
- 45. Route Tables • Route Table (rtb) = Multiple sets of routes • Default or Main Route Table is nothing specific is configured • Each Subnet is associated with a single Route Table • Routes are either static (manually configured) or propagated (from BGP connections) • Priority • Most specific match • Static • Propagated 45
- 46. Route Tables • Route in a Route Table = Prefix + Next Hop • Next Hop Types • Instance NIC (Src/Dst Check) • Peering Connection: For connections to other VPCs • Virtual Private Gateway: For connections to VPNs or Private Circuits • NAT Gateway: For egress NATing • VPC Endpoint: For supported AWS Services • Internet Gateway 46
- 47. System Routes + BGP + UDR • All Subnets have System Routes • Can't be removed, but can be shadowed by custom routes • Can add additional (custom) User Defined Routes via a Route Table • Priority • Most specific match • User Defined Route • BGP System Route • System Internet Route 47
- 48. System Routes • 3 Default Route Sets automatically associated with VNet • Local VNet's Subnets • Associated Networks: Routes propagated from Peering / VPN / ExpressRoute • Internet 48
- 49. User Defined Routes • Customer configured routes that exist additionally to System Routes • Prefix + Next Hop. Next Hop one of: • (Local) VNet: For VNet CIDR Destinations • Virtual Network Gateway: Site-to-Site Connection • Virtual Appliance: VMs inside VNet (IP Forwarding) • Internet • None: Blackhole • Can't route back into a subnet 49
- 50. Routes • Global Routing Table for a VPC Network • Custom Static Routes • Priority • Most specific match • Highest priority (lowest by value) • Multipath (Hash: Protocol, Src IP, Dst IP, Src Port, Dst Port) 50
- 51. Routes • Route = • Name + VPC network + Prefix + (optional) Instance Tags + Priority + Next Hop. • Next Hop one of • Instance • IP - inside of network (must be a primary IP) • Gateway - The internet gateway (though maybe more options in the future) • VPN Tunnel 51
- 52. AWS Azure GCP Name Route Tables System Routes + User Defined Routes Routes Route Selection Most specific CIDR match, Static Propagated Most Specific CIDR match, User Defined Routes, BGP Routes, System Routes Most specific CIDR match, then by priority, then mutlipath hash Route Sharing All subnets associated with same Route Table All subnets associated with same User Defined Route One shared route table; Specific route rules applied via instance tag Default Routes for New VPC CIDR Internet Peered Connections VNet CIDR Internet Subnet CIDRs
- 53. Use an Instance as a Gateway • Connect two different network areas together • Use an instance as a gateway 53
- 55. Ingress/Egress • For Instance to get in/out, it has to have • Route to Internet ("Route - Inside") • Policy to permit access to/from Internet ("Access Control") • Allocated Public IP on the CSP's external network • Or - something else can do the work for it 55
- 56. CSP Public IP Selection • All Internet routed addresses come from the CSP's allocated Internet IP space • Allocated to a Region • No BYOIP • Two types • Ephemeral Addresses: allocated and released when VM starts/terminates • Static Addresses: allocated ahead of time and remain attached to the Account even if the VM is terminated (called Elastic IP for AWS) • Azure/GCP: Can promote an ephemeral public IP to static 56
- 57. Public IP == Public NAT No Public IP == Possibly Only Egress Traffic* • If a VM has a Public IP (ephemeral or static) associate with it, it gets a NAT for Internet traffic • The remainder of this section talk mostly about the "No Public IP" cases 57
- 58. Internet Gateway (IGW) • No default Internet Route • Have to allocate an IGW to the VPC • Have to add route to IGW • Instance has to be configured with a public IP 58
- 59. NAT Gateway (NGW) • AWS Managed Service which provides SNAT for egress traffic only • Must provide AWS with designated Elastic IP (public side) and subnet which has an internet route • Other subnets must use a different Route Table to send 0/0 to the NAT Gateway • Assigned to an Availability Zone • But can support multiple AZs --- impact on resilience • Common patterns to see one ngw per AZ • Dependent on an IGW+Route for the NGW's outside access 59
- 60. Shared SNAT • Default Internet Route provided • Default shared SNAT • Shared == possibly with other accounts • Prevent this with Access Control (NSG) 60
- 61. Ephemeral IP • Default Internet Route provided • Default ephemeral public IP provided on each instance • can choose not to allocate • can be limited with Organization Policy (beta) 61
- 62. NAT? • No Managed Service • You stand up your own instance which is performing NAT • Add 0/0 to override routes • Can use tags to decide which Instances get external access • Can use tags to decide which Instances get NAT access 62
- 63. AWS Azure GCP Default Internet Route for New Net None Yes Yes Default NAT None SNAT Ephermeral IP
- 64. DMZ + Private • Want DMZ hosts which can get out, and Internal hosts which have to go through the DMZ to get out 64
- 65. Routing - CSP Networks 65
- 66. Routing - CSP Networks • On their networks, CSPs have... • Other Network Substrates that we may want to privately communicate with • Other Cloud Services • These services may exist in local or remote Regions • CSPs have a lot of similar behaviors - differences highlighted 66
- 67. VPC Peering (pcx) • Private connectivity between two VPCs in the same Region • Same or different accounts • Latency/bandwidth/cost is same as talking inside a VPC • Non-overlapping IP space; Non-transitive • Add routes to the Route Table(s) to be connected • Can reference Security Groups in foreign VPC 67
- 68. VPC Endpoints (vpce) • Private connectivity between VPC and AWS Services • Limited to same region services only • Non-transitive • Only some services provide VPC Endpoints - S3, DynamoDB • Given an identifier (pl-xxxx) to use in Route Tables • In some service policies, can use VPC as a permission item (i.e. this VPC can access this service) • Can reference VPC Endpoint (pl-xxxx) in Security Groups (but not NACL) 68
- 69. Virtual Network Peering • Private connectivity between two VNets in the same Region • Can be different Subscriptions but same Account* • Non-overlapping IP space; Non-transitive, but can share VPN Gateway • Latency/bandwidth/cost is same as talking inside a VPC • System Routes are automatic for Peered VNet • Can override with UDR to point to instances (on either side) as gateway 69 * Being worked on, but in the interim, use VPN peering
- 70. VPC Network Peering • Private connectivity between VPC Networks (in all Regions) • Same or different projects/organizations • Latency/bandwidth/cost is same as talking inside a VPC • Non-overlapping IP space; Non-transitive • Once peered, can't create conflicts (overlapping subnets, routes overlapping, etc) among the full set of VPC Networks that are peered (even between two that aren't direct) 70
- 71. Private Google Access • Private connectivity (IP space exposed) to Google Services • Can be enabled on a Subnet basis • Must still use internet route to access (can be tag limited) 71
- 72. AWS Azure GCP Peering Name VPC Peering VNet Peering VPC Network Peering Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations Limits 50/125 Peers 10/50 Peers 25 Peers 7500 Instances Combined CSP Service Peering Name VPC Endpoint N/A Private Google Access
- 73. Peers + Cloud Services • Peer Two Networks in the same CSP (same Account/Subscription/Project) • Allow (preferably) private connections from both networks to CSP Cloud Services 73
- 74. Routing - Private Gateways 74
- 75. Private Gateways • Connection points for linking CSP Network to an outside CSP network • VPN • Physical • Can be managed by a third party - Cloud Exchanges • In all cases, you can run your own VPN software (third party VM appliances) 75
- 76. CSP Managed VPN • Use of IPSec devices to connect privately to an on-premise (or other) network • Connect VPC/VNet/VPC Network to Corporate Office or Data Center • Connect VPC/VNet/VPC Network to other CSP Network • Inter-Region VPC/VNet network connectivity 76
- 77. Physical • Private Circuit connecting your network at a CSP POP to an associated CSP Region • If not adjacent to POP, have to get tail circuit to connect your site to POP • Can also be used to peer with Public side of CSP network • Access to other CSP services • Access to CSP management interfaces 77
- 78. Limits • VPNs and Physical Connections have limits • In general, can setup parallel connections and rely on multipath • Multiple VPN tunnels working together • Multiple Physical connections working together • Combination of Physical and VPN - preference usually goes to Physical 78
- 79. Virtual Private Gateway (VGW) • Logical router sitting outside of VPC • Associated with single VPC; so limited to a Region • All foreign networks come in via VGW • Set routes in the VPC Route Tables to forward traffic to VGW • Or allow routes to be propagated from the VGW peers • Only the VPC CIDR is advertised to VGW and its peers • Can't transit for VPC Peering or VPC Endpoints or Internet traffic - though can proxy • Will transit for non-VPC peers - "CloudHub" 79
- 80. Hardware VPN • Private IPSec connectivity between AWS and non-AWS managed VPN device (aka Customer Gateway - CGW) • Can't use between two AWS Regions • Site-to-Site • Always builds two tunnels (two devices are allocated on the AWS side) to one CGW • If you want redundancy on the CGW side, you'll build four tunnels • IP of AWS VPN side not designated until VPN Connection is configured • Must supply static IP for CGW (can use same for multiple connections - NAT-T supported) • Static or BGP 80
- 81. Direct Connect (DX) • AWS Physical Connection (CON) • 1Gbps, 10Gbps options (can do smaller through a Cloud Exchange) • Can also use multiple (<=4) CON together with LACP • BGP Peering Only 81
- 82. Direct Connect (DX) - Sharing • Can be used with multiple accounts/VPCs. • Each Direct Connect Connection (CON) is split into virtual interfaces (VIF). • You assign the VIF information • VLAN Assignment • BGP Neighbor • Hosting Account (one which has the CON) handles creation of VIF • Guest Account attaches VIF to VGW • Once attached, can't reattach (have to destroy/recreate) 82
- 83. Virtual Network Gateway • Managed Gateway Devices • Deployed as VMs inside of a dedicated "Gateway Subnet" (/27 or larger) • Two types • VPN Gateway • ExpressRoute • Can only have one of each type per VNet • Multiple SKUs for sizing 83
- 84. VPN Gateway • Site-to-Site • IPSec from VPN Gateway to Local Network Gateway • Allocated to a Region • Static (Policy list of CIDRs) or BGP Route Based • Limits/Sizing place preference on Route Based • Gateway Transit: can extend reachability to Peered VNets • Use this topology to get around some limits 84
- 85. VPN Gateway • VNet-to-VNet • (special case of Site-to-Site) • Connect separate Regions and separate Accounts/Subscriptions together • Point-to-Site • Access via the in-box Windows VPN SSTP Client 85
- 86. ExpressRoute • Private BGP Connections to Microsoft • Handoff • IP VPN (i.e. MPLS) • Ethernet tail to POP • Cloud Exchange Virtual Cross Connection • Handoff affects location of on-prem and latency, but not necessarily Region access (depending on Subscription level) • Offered as redundant pair of connections (no SLA without) • Provides • VNet Private Peering • Azure Public Peering • (add-on) Microsoft Peering for Office 365 et al 86
- 87. ExpressRoute • For each Circuit, can setup one to all of the three peering types • Peering Information: • BGP IP/Neighbor IP • VLAN • Neighbor ASN - Public restrictions for Public/Microsoft • For Microsoft: Routing Registry Name • Restrictions on Peering Type for advertised routes (# and RFC1918) 87
- 88. ExpressRoute to VNet • Owner Subscription handles Circuit creation and Peering configuration • Owner Authorizes Other Subscriptions • Other Subscription link VNets to ExpressRoute Circuit VNet Peering • Must have a Virtual Network Gateway (ExpressRoute Type) configured ahead of time • Standard vs Premium limits on # of VNet linkages and Region • All VNets linked to the same ExpressRoute Circuit • ExpressRoute and Point-to-Site are not supported together in the same VNet 88
- 89. Cloud Router • Managed Router which handles all dynamic routing via BGP relationships with other routers • Makes changes to your Routes • Scoped to local or global route changes (i.e. advertise remote Regions) • You assign a private ASN to Cloud Router • Create multiple interfaces (link-local addressing) • Create multiple BGP neighbor configurations • Unlike the other CSPs, even if you setup tunnel devices, the Cloud Router is the BGP neighbor 89
- 90. Cloud VPN Gateway • Managed IPSec VPN Endpoint which can run multiple tunnels • Allocated to a Region, but can forward any traffic on the VPC Network • Can connect too another Project's or Organization's VPN Gateway • You specify tunnels to create (so not always in pairs - but that is recommended) • Typical Tunnel Configuration • Local IP: Assigned from your pool of static external IPs • (Static mode) Local subnets, IP ranges - Can't change after creating • Peer IP, IKE, ESP, PFS, Shared Secret • NAT-T not supported 90
- 91. Cloud VPN Gateway • Static Mode • Specify traffic selection via --remote-traffic-selector • Add VPN Gateway as a NextHop in your route table • Dynamic Mode • Create an interface on Cloud Router that is associate to the tunnel • Handle all traffic selection via BGP relationship with Cloud Router • Cloud Router adds VPN Gateway as NextHop • BGP Peer Information can be configured or have GCP generate it 91
- 92. Dedicated Interconnect • GCP Physical Connection • 10 Gbps • Can use multiple with LACP • BGP Established with Cloud Router (i.e. requires Cloud Router) • Is divided up into VLAN Attachments • GCP allocates the VLAN and BGP IPs to use (ASN specified by Cloud Router) • Use that information to configure Cloud Router and on-prem router 92
- 93. Dedicated Interconnect - Sharing • Can be shared among Projects in an Organization • Even if not using a Shared VPC Network • Two methods - both using IAM permissions • Hosting Project grants permissions to Guest Project's Users to update Interconnect • Guest Project grants permissions to Hosting Project to update Gust Project's Cloud Router 93
- 94. AWS Azure GCP Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router Gateway Model Router outside of VPC Is NextHop for all gateway traffic VM Insides of "Gateway Subnet" which handle routing and gateway BGP Talker that updates routes. Separate gateways that forward traffic. Substrate Route Mgmt You add routes to RTB, or setup to propagate all routes from VGW to RTB Routes show up as System Routes Cloud Router updates Routes Gateway Peer Transit Yes VPN: Yes including VNet Peering ExpressRoute: No Yes
- 95. AWS Azure (VNet Peering) GCP VPN Name VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway Cloud Interconnect - IPSec VPN aka Cloud VPN Options Site-to-Site Point-to-Site Site-to-Site Site-to-Site Deploy Model Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop Endpoint associated with VPC Network as NextHop VPN Transit Yes (CloudHub) No Yes Size Options 1 3 (4) 1 Advertised Routes 1 Prefix 1 Prefix Static: 128 Prefixes BGP: 100 Prefixes (CR limit) Received Route limits 100 Prefixes . 100 Prefixes (CR limit) Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway Route Selection Static: What is specified BGP: VPC + All VGW BGP Peers Static: What is specified BGP: VNet, Configured Peers Static: What is specified BGP: By Cloud Router
- 96. AWS Azure GCP Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated Interconnect Link Options 1Gbps, 10Gbps, LACP IP VPN 50,100,200,500M ; 1,2,5,10Gbps Cloud Exchange 10Gbps, LACP Routing BGP BGP BGP (to Cloud Router) VLAN/BGP Allocation You Pick You Pick GCP Picks Sharing Yes Centrally Managed Yes Centrally Managed Configuration RBAC VNet Linking Yes RBAC Managed Route Limits 100 Prefixes Depends on Size VNet: 4000-10000 100 (CR limit)
- 97. CSP Network as Data Center Extension • Setup CSP Network which only has connectivity from a Private Data Center via a physical connection • Any ingress or egress traffic goes via the Private Data Center 97
- 98. Cross Region Network • Want to peer two Networks in the Same VPC 98
- 99. Connect Multiple CSP Together • Have two CSP Networks • One in each of two different CSPs • "Peer" them together 99
- 100. Cloud Storage from Corp Office • Setup protected path from corporate office to Storage offering of CSP 100
- 101. Access Control 101
- 102. AWS Access Control • Two types • Security Groups (SG): For instances • Network Access Control Lists (NACLs): For subnets 102
- 103. Security Group (SG) • Stateful Packet Filters • Separate lists for ingress and egress • Applied on Network Interfaces • By "Applied to Instance," it really means "Applied on the Instance's Primary NIC" • Positive Control: Only ALLOW. No DENY • Can apply multiple (5) SG on an NIC • ALLOWs are cumulative 103
- 104. Security Group (SG) • Structure: • Protocol • Port Range, or ICMP Type • Source (ingress), or Destination (egress) CIDR, or SG • New SG has default egress allows anything out • "default" SG available when VPC created (and always available) • Ingress: ALLOW from "default" SG to ANY • Egress: ALLOW to ANY 104
- 105. Security Group (SG) • Applied to Instances • Context is focused on Instance (or groups of Instances) Rules • But with tagging is hard to tailor to only your instances for modifying SG 105
- 106. Network Access Control List (NACL) • Stateless • Only one allowed per subnet • Ordered priority • Positive and Negative control (ALLOW/DENY) 106
- 107. Azure Access Control • Just one • Network Security Groups (NSG) 107
- 108. Network Security Group (NSG) • 1 per NIC and/or subnet • Applied in priority order (ascending) • default rules for inbound and outbound on both (permit out, permit VNET) • The NSG on the NIC, if present, and on the Subnet have to permit to permit traffic. One DENY on either will block traffic. • NSG Tags : canned collections of CIDRs (e.g. VIRTUAL_NETWORK) • Can do flow logging 108
- 109. Network Security Group (NSG) • Structure • Priority (Low numerically is Higher Priority) • Name • Protocol + Source IP/Tag + Source Port + Destination IP/Tag + Destination Port • Allow/Deny 109
- 110. Network Security Group (NSG) • Managing • Action (Permissions): Microsoft.Network/networkSecurityGroup/* • Roles: Network Contributor, Owner • Context is global to VNet 110
- 111. Firewall • Single managed firewall for VPC network • Manages inbound and outbound • Can limit sources/destinations using Target Tags • Tag matched against Tags on Instance 111
- 112. Firewall • Structure • Priority (ascending, first match wins) • (Ingress) Protocol + Destination Port + Source IP • (Egress) Protocol + Destination Port + Destination IP • Target Tags • Allow/Deny 112
- 113. Firewall • Managing • Permissions: compute.firewalls.* • Role: roles/compute.securityAdmin • Context is global to Project/Share VPC Project 113
- 114. AWS AWS Azure GCP Name Security Group Network Access Control List Network Security Group Firewall State Stateful Stateless Stateful Stateful Additive Positive Only Positive/Negative Postive/Negative Positive/Negative Target NIC Subnet NIC or Subnet VPC (target tags for instance) Multiple Yes (5 on instance 250 in account) No No No Ordered No Yes Yes Yes
- 115. Setup Access Control for 3 Tier App • Given 4 VMs - web, app, db, bastion - setup access control such that: • Anyone is allowed from any tcp port to "web" on 443/tcp • "web" is allowed from any tcp port to "app" on 8009/tcp • "app" is allowed from any tcp port to "db" on 3306/tcp • "bastion" is allowed from any tcp port to all three instances on 22/tcp • "bastion" is allowed from any udp port to all three instances on 161/udp 115
- 116. Summary and Next Steps 116
- 117. Locations 117 Regions AWS 16 44 Availability Zones Azure 26 (not counting Gov) 60 Fault Domains (not counting Gov) GCP 12 36 Zones
- 118. AWS Azure GCP Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network IP Addressing RFC1918 or Other Carving up CIDR of VPC RFC1918 or Other Carving up CIDR of VNet RFC1918 only Accumulation of Subnet CIDRs Locality One Region One Region Global Subnet Locality One AZ Region Selection of Zones in a Region CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR Account Resource Sharing No Users use multiple Subs. Resources bound to one VNet inside one Sub at a time Across Projects - Yes Across Organizations - No Substrate
- 119. AWS Azure GCP Forwarding Property Source/Destination Check IP Forwarding (enableIPForwarding) IP Forwarding (can-ip-forward) Property Default On Off Off NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface IPs per NIC 6-50 50 Unspecified (Alias IP not supported with multiple NICs) NICs per Instance 1-15 2-8 1-8 NIC Locations Same VPC Same VNet Each must be on separate VPC Networks Instance Properties
- 120. AWS Azure GCP Name Route Tables System Routes + User Defined Routes Routes Route Selection Most specific CIDR match, Static Propagated Most Specific CIDR match, User Defined Routes, BGP Routes, System Routes Most specific CIDR match, then by priority, then mutlipath hash Route Sharing All subnets associated with same Route Table All subnets associated with same User Defined Route One shared route table; Specific route rules applied via instance tag Default Routes for New VPC CIDR Internet Peered Connections VNet CIDR Internet Subnet CIDRs Routing - Inside
- 121. AWS Azure GCP Default Internet Route for New Net None Yes Yes Default NAT None SNAT Ephermeral IP Routing - Internet
- 122. AWS Azure GCP Peering Name VPC Peering VNet Peering VPC Network Peering Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations Limits 50/125 Peers 10/50 Peers 25 Peers 7500 Instances Combined CSP Service Peering Name VPC Endpoint N/A Private Google Access Routing - CSP Networks
- 123. AWS Azure GCP Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router Gateway Model Router outside of VPC Is NextHop for all gateway traffic VM Insides of "Gateway Subnet" which handle routing and gateway BGP Talker that updates routes. Separate gateways that forward traffic. Substrate Route Mgmt You add routes to RTB, or setup to propagate all routes from VGW to RTB Routes show up as System Routes Cloud Router updates Routes Gateway Peer Transit Yes VPN: Yes including VNet Peering ExpressRoute: No Yes Routing - Private Routes
- 124. AWS Azure (VNet Peering) GCP VPN Name VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway Cloud Interconnect - IPSec VPN aka Cloud VPN Options Site-to-Site Point-to-Site Site-to-Site Site-to-Site Deploy Model Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop Endpoint associated with VPC Network as NextHop VPN Transit Yes (CloudHub) No Yes Size Options 1 3 (4) 1 Advertised Routes 1 Prefix 1 Prefix Static: 128 Prefixes BGP: 100 Prefixes (CR limit) Received Route limits 100 Prefixes . 100 Prefixes (CR limit) Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway Route Selection Static: What is specified BGP: VPC + All VGW BGP Peers Static: What is specified BGP: VNet, Configured Peers Static: What is specified BGP: By Cloud Router Routing - VPN
- 125. AWS Azure GCP Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated Interconnect Link Options 1Gbps, 10Gbps, LACP IP VPN 50,100,200,500M ; 1,2,5,10Gbps Cloud Exchange 10Gbps, LACP Routing BGP BGP BGP (to Cloud Router) VLAN/BGP Allocation You Pick You Pick GCP Picks Sharing Yes Centrally Managed Yes Centrally Managed Configuration RBAC VNet Linking Yes RBAC Managed Route Limits 100 Prefixes Depends on Size VNet: 4000-10000 100 (CR limit) Routing - On-Premise
- 126. AWS AWS Azure GCP Name Security Group Network Access Control List Network Security Group Firewall State Stateful Stateless Stateful Stateful Additive Positive Only Positive/Negative Postive/Negative Positive/Negative Target NIC Subnet NIC or Subnet VPC (target tags for instance) Multiple Yes (5 on instance 250 in account) No No No Ordered No Yes Yes Yes Access Control
- 127. Additional "Network" Areas • Load Balancers - some affect SNAT and packet forwarding • DNS - interacts with Substrate DNS view • CDN 127
- 128. Limits!!! • http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/ VPC_Appendix_Limits.html • https://docs.microsoft.com/en-us/azure/azure-subscription-service- limits#networking-limits • https://cloud.google.com/router/quotas 128
- 129. October 29–November 3, 2017 | San Francisco, CA www.usenix.org/lisa17 #lisa17 Remember to fill in your tutorial evaluation! Thank You! R6 - The Ins-and-Outs of Networking in the Big Three Clouds Chris "mac" McEniry