SlideShare a Scribd company logo

Data Center Network Topologies

•
R
rjain51

The document discusses data center network topologies, focusing on physical layouts, modular designs, and the hierarchical structures of modern data center networks. It highlights issues such as cabling, service delivery, and network efficiency, including the advantages of using fat-tree topologies. Key requirements for data center networks include scalability, security, and support for large Layer 2 domains for effective workload mobility.

TechnologySpiritual
1 of 35
Downloaded 982 times
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
14
Most read
15
16
17
18
19
Most read
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Data Center Network Topologies . Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides and audio/video recordings of this class lecture are at: http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-1 ©2013 Raj Jain
Overview 1. 2. 3. 4. 5. Data Center Physical Layout Data Center Network Topologies ToR vs. EoR Data Center Networking Issues Data Center Networking Requirements Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-2 ©2013 Raj Jain
Google’s Data Center Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-3 ©2013 Raj Jain
Cooling Plant Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-4 ©2013 Raj Jain
Modular Data Centers      Small: < 1 MW, 4 racks per unit Medium: 1-4 MW, 10 racks per unit Large: > 4 MW, 20 racks per unit Built-in cooling, high PUE (power usage effectiveness) 1.02 PUE = Power In/Power Used Rapid deployment Ref: http://www.sgi.com/products/data_center/ice_cube_air/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-5 ©2013 Raj Jain
Containerized Data Center     Ready to Use. Connect to water and power supply and go. Built in cooling. Easy to scale.  Data Center trailer parks. Suitable for disaster recovery, e.g., flood, earthquake Offered by Cisco, IBM, SGI, Sun/ORACLE,… Ref: Datacenter Infrastructure – mobile Data Center from Emerson Network Power , http://en.m-info.ua/180-container-data-center/755-datacenter-infrastructure-mobile-data-center-from-emerson-network-power Ref: http://www.datacenterknowledge.com/archives/2010/05/31/iij-will-offer-commercial-container-facility/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-6
Unstructured Cabling Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-7 ©2013 Raj Jain
Structured Cabling Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-8 ©2013 Raj Jain
Data Center Equipment Cabinets Three Layers: Bottom: Signal, Middle: Power, Top: Fiber Minimize patching between cabinets and racks Cabling under raised floors provides better appearance and cooling Ref: Ref: C. DiMinico, "Telecommunications Infrastructure Standard for Data Centers," IEEE 802.3 HSSG Meeting, Nov. 2006, http://www.ieee802.org/3/hssg/public/nov06/diminico_01_1106.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-9
Data Center Physical Layout Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-10 ©2013 Raj Jain
ANSI/TIA-942-2005 Standard     Main Distribution Area (MDA) Horizontal Distribution Area (HDA) Equipment Distribution Area (EDA) Zone Distribution Area (ZDA) Source: Santana 2014 Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-11 ©2013 Raj Jain
ANSI/TIA-942-2005 Standard Computer Room: Main servers  Entrance Room: Data Center to external cabling  Cross-Connect: Enables termination of cables  Main Distribution Area (MDA): Main cross connect. Central Point of Structured Cabling. Core network devices  Horizontal Distribution Area (HDA): Connections to active equipment.  Equipment Distribution Area (EDA): Active Servers+Switches. Alternate hot and cold aisle.  Zone Distribution Area (ZDA): Optionally between HDA and EDA. ZDA allows easy  Backbone Cabling: Connections between MDA, HDA, and Entrance room http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain  3-12
Zone Distribution Area  High-fiber count cables connect ZDA to MDA or HDA. Low-fiber count cables connect ZDA to EDA as needed. Ref: Jennifer Cline, “Zone Distribution in the data center,” http://www.graybar.com/documents/zone-distribution-in-the-data-center.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-13 ©2013 Raj Jain
Data Center Network Topologies  Core, Aggregation, Access Ref: G. Santana, “Data Center Virtualization Fundamentals,” Cisco Press, 2014, ISBN:1587143240 http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-14 ©2013 Raj Jain
Data Center Networks        20-40 servers per rack Each server connected to 2 access switches with 1 Gbps (10 Gbps becoming common) Access switches connect to 2 aggregation switches Aggregation switches connect to 2 core routers Core routers connect to edge routers Aggregation layer is the transition point between L2-switched access layer and l3-routed core layer Low Latency: In high-frequency trading market, a few microseconds make a big difference.  Cut-through switching and low-latency specifications. Ref: A. Greenberg, "VL2: A Scalable and Flexible Data Center Network," CACM, Vol. 54, NO. 3, March 2011, pp. 95-104, http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf. http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-15
Data Center Networks (Cont)       Core routers manage traffic between aggregation routers and in/out of data center All switches below each pair of aggregation switches form a single layer-2 domain Each Layer 2 domain typically limited to a few hundred servers to limit broadcast Most traffic is internal to the data center. Network is the bottleneck. Uplinks utilization of 80% is common. Most of the flows are small. Mode = 100 MB. DFS uses 100 MB chunks. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-16 ©2013 Raj Jain
Switch Locations Smaller cable between servers and switches Network team has to manage switches on all racks All network switches in one rack Source: Santana 2014 Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-17 ©2013 Raj Jain
ToR vs EoR  ToR:  Easier cabling    If rack is not fully populated  unused ToR ports If rack traffic demand is high, difficult to add more ports Upgrading (1G to 10G) requires complete Rack upgrade   EoR:  Longer cables  Severs can be place in any rack  Ports can easily added, upgraded Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-18 ©2013 Raj Jain
Hierarchical Network Design     All servers require application delivery services for security (VPN, Intrusion detection, firewall), performance (load balancer), networking (DNS, DHCP, NTP, FTP, RADIUS), Database services (SQL) ADCs are located between the aggregation and core routers and are shared by all servers Stateful devices (firewalls) on Aggregation layer Stateful= State of TCP connection Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-19 Source: Santana 2014 ©2013 Raj Jain
Access Aggregation Connections 1. 2. 3. 4. Looped Triangle: Most common. Spanning Tree Protocol (STP) blocks links. Paid but not used. Looped Square: Oversubscription doubles if failure. Loop-Free U: No L2 communication between aggregation switches if any switch links fail Loop-Free Inverted U: Black-holes on some failures Washington University in St. Louis Source: Santana 2014 http://www.cse.wustl.edu/~jain/cse570-13/ 3-20 ©2013 Raj Jain
Data Center Networking Issues     Higher layers oversubscribed:  Other servers in the same rack 1:1  Uplinks from ToR: 1:2 to 1:20 (e.g., 32x10Gb down, 8X10Gb up  4:1 oversubscription)  Core Routers: 1:240  Generally keep services in one tree  Can't arbitrarily move servers Moving across Subnets is painful  Requires reconfiguration of IP addresses and VLAN trunks Service trample on each-other. Overuse by one service affects others Poor reliability. One access switch failure doubles the load on the other. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-21 ©2013 Raj Jain
Data Center Networking Issues (Cont)   Under-utilization. Even when multiple paths exist only one is used. ECMP (Equal Cost Multipath) is used by routers to spread traffic to next hops using a hash function. However, only 2 paths exist. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-22 ©2013 Raj Jain
DCN Requirements  Needs to be Scalable, Secure, Shared, Standardized, and Simplified (5 S's)  Converged Infrastructure: Servers, storage, and network have to work together Workload Mobility: Large L2 domains required for VM mobility East-West Traffic: Significant server-to-server traffic as compared to server to user. One Facebook request required 88 cache looks, 35 database lookups, 392 backend RPC calls. Internet traffic 935X the http request/response [Farrington] Storage traffic on Ethernet: Congestion management on Ethernet    Ref: A. Kindlness, "The Forester Wave: Data Center Networking Hardware," Jan 23, 2013, http://ca.westcon.com/documents/46488/forrester_wave_data_cetner_networking_hw_q1_2013.pdf Ref: N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-23
4-Post Architecture at Facebook     Each rack switch (RSW) has up to 48 10G downlinks and 4-8 10G uplinks (10:1 oversubscription) to cluster switch (CSW) Each CSW has 4 40G uplinks – one to each of the 4 FatCat (FC) aggregation switches (4:1 oversubscription) 4 CSW’s are connected in a 10G×8 protection ring 4FC’s are connected in a 10G×16 protection ring No routers at FC. One CSW failure reduces intra-cluster capacity to 75%. Ref: N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-24
Clos Networks        Multi-stage circuit switching network proposed by Charles Clos in 1953 for telephone switching systems Allows forming a large switch from smaller switches The number of cross-points is reduced  Lower cost (then) 3-Stage Clos(n, m, r): ingress (r n×m), middle (m r×r), egress (r m×n) Strict-sense non-blocking if m > 2n-1. Existing calls unaffected. Rearrangeably non-blocking if m > n Can have any odd number of stages, e.g., 5 Folded: Merge input and output in to one switch = Fat-tree n×m n×m r=2, n=4, m=3 r×r r×r r×r Ref: http://en.wikipedia.org/wiki/Clos_network Washington University in St. Louis r×r m×n r×r m×n r×r n+m port switches r-port switches http://www.cse.wustl.edu/~jain/cse570-13/ 3-25 ©2013 Raj Jain
Fat-Tree DCN Example Core 9 Edge Servers     … … … … 6 identical 36-port switches. All ports 1 Gbps. 72 Servers. Each edge switch connects to 18 servers. 9 Uplinks to first core switch. Other 9 links to 2nd core switch. Throughput between any two servers = 1 Gbps using ECMP Identical bandwidth at any bisection. Negative: Cabling complexity Ref: Teach yourself Fat-Tree Design in 60 minutes, http://clusterdesign.org/fat-trees/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-26 ©2013 Raj Jain
Summary 1. 2. 3. 4. 5. Modular data centers can be used for easy assembly and scaling Three tiers: Access, Aggregation, Core Application delivery controllers between Aggregation and core Need large L2 domains Fat-tree topology is sometimes used to improve performance and reliability Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-27 ©2013 Raj Jain
Homework 3  Draw a 3-stage clos(4,5,3) topology and its folded version. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-28 ©2013 Raj Jain
Acronyms ADC ANSI BPE CSW DCBX DCN DFS DHCP DNS ECMP EDA EoR Application Delivery Controller American National Standards Institute Business Process Engineering Core Switch Data Center Bridging eXtension Data Center Network Distributed File System Dynamic Host Control Protocol Domain Name System Equal Cost Multipath Equipment Distribution Area End of Row Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-29 ©2013 Raj Jain
Acronyms (Cont) ETS Enhanced Transmission Selection EVB Edge Virtual Bridge FC Fibre Channel FSW Fabric switch FTP File Transfer Protocol HDA Horizontal Distribution Area LACP Link Aggregation Control Protocol LAG Link Aggregation LLDP Link Layer Discovery Protocol MAC Media Access Control MDA Main Distribution Area MW Mega-Watt NTP Network Time Protocol Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-30 ©2013 Raj Jain
Acronyms (Cont) NVGRE Network Virtualization using Generic Routing Encapsulation PFC Priority Flow Control PUE Power Usage Effectiveness RADIUS Remote Authentication Dial-In User Service RPC Remote Procedue Call RSW Rack switch SQL Structured Query Language SSW Spine Switches STP Spanning Tree Protocol TIA Telecommunications Industry Association ToR Top of Rack TRILL Transparent Interconnection of Lots of Link VLAN Virtual Local Area Network VM Virtual Machine VPN Virtual Private Network Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-31 ©2013 Raj Jain
Acronyms (Cont) VRF Virtual Routing and Forwarding VXLANVirtual Extensible Local Area Network ZDA Zone Distribution Area Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-32 ©2013 Raj Jain
Reading List        http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ http://www.sgi.com/products/data_center/ice_cube_air/ Datacenter Infrastructure - mobile Data Center from Emerson Network Power, http://www.datacenterknowledge.com/archives/2010/05/31/iij-willoffer-commercial-container-facility/ C. DiMinico, "Telecommunications Infrastructure Standard for Data Centers," IEEE 802.3 HSSG Meeting, Nov. 2006, http://www.ieee802.org/3/hssg/public/nov06/diminico_01_1106.pdf Jennifer Cline, “Zone Distribution in the data center,” http://www.graybar.com/documents/zone-distribution-in-the-datacenter.pdf G. Santana, “Data Center Virtualization Fundamentals,” Cisco Press, 2014, ISBN:1587143240 (Safari book) A. Greenberg, "VL2: A Scalable and Flexible Data Center Network," CACM, Vol. 54, NO. 3, March 2011, pp. 95-104, http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-33 ©2013 Raj Jain
Reading List (Cont)      A. Kindlness, "The Forester Wave: Data Center Networking Hardware," Jan 23, 2013, http://ca.westcon.com/documents/46488/forrester_wave_data_cetner_netwo rking_hw_q1_2013.pdf N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://en.wikipedia.org/wiki/Clos_network Teach yourself Fat-Tree Design in 60 minutes, http://clusterdesign.org/fattrees/ M. Al-Fares, et al, "A scalable, commodity data center network architecture," ACM SIGCOMM, 2008. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-34 ©2013 Raj Jain
Wikipedia Links          http://en.wikipedia.org/wiki/Modular_data_center http://en.wikipedia.org/wiki/Data_center http://en.wikipedia.org/wiki/Structured_cabling http://en.wikipedia.org/wiki/Cable_management http://en.wikipedia.org/wiki/Raised_floor http://en.wikipedia.org/wiki/Data_center_environmental_contr ol http://en.wikipedia.org/wiki/Fat_tree http://en.wikipedia.org/wiki/Hierarchical_internetworking_mod el http://en.wikipedia.org/wiki/Clos_network Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-35 ©2013 Raj Jain

More Related Content

PPT
Data center
Mohammad Danish
 
PPTX
Data Center
dhana1663
 
PPTX
Data centers
tejaswi25
 
PPTX
Data center
Jwala Chapagai
 
ODP
Datacenter101
tarundua
 
PPTX
Data center proposal
Muhammad Ahad
 
PDF
Data Center Checklist for Infrastructure Best Practices (SlideShare)
SP Home Run Inc.
 
PPTX
Data Center Infrastructure Management(DCIM)
MD. IFTEKARUL ALAM
 
Data center
Mohammad Danish
 
Data Center
dhana1663
 
Data centers
tejaswi25
 
Data center
Jwala Chapagai
 
Datacenter101
tarundua
 
Data center proposal
Muhammad Ahad
 
Data Center Checklist for Infrastructure Best Practices (SlideShare)
SP Home Run Inc.
 
Data Center Infrastructure Management(DCIM)
MD. IFTEKARUL ALAM
 

What's hot (20)

PDF
SD WAN Overview | What is SD WAN | Benefits of SD WAN
Ashutosh Kaushik
 
PPTX
POWER POINT PRESENTATION ON DATA CENTER
vivekprajapatiankur
 
PPTX
Data Center Networks
Sayed Chhattan Shah
 
PDF
SDWAN vs MPLS: What Enterprises need?
Haris Chughtai
 
PPTX
Software Defined Network - SDN
Venkata Naga Ravi
 
PPTX
Datacenter overview
Heather Brotherton
 
ZIP
DataCenter:: Infrastructure Presentation
Muhammad Asad Rashid
 
PPTX
SD WAN
Bri Molina
 
PPTX
Firewall presentation
TayabaZahid
 
PPTX
Introduction to Data Center Network Architecture
Ankita Mahajan
 
PPTX
Router Full Concept pdf
Vignesh kumar
 
PDF
Network Address Translation (NAT)
Joud Khattab
 
PDF
ClearPass Overview
JoAnna Cheshire
 
PPTX
Customer Presentation - Aruba Wi-Fi Overview (1).PPTX
ssuser5824cf
 
PPTX
Real-world 802.1X Deployment Challenges
Aruba, a Hewlett Packard Enterprise company
 
PPTX
Data Centre Design Guideline and Standards
Bangladesh Network Operators Group
 
PDF
Data Center Infrastructure Management Powerpoint Presentation Slides
SlideTeam
 
ODP
Private Cloud Architecture
Derek Keats
 
PPTX
Overview of Major Aruba Switching Features incl. Smart Rate for Multi-Gig Ports
Aruba, a Hewlett Packard Enterprise company
 
PPT
Fortigate Training
NCS Computech Ltd.
 
SD WAN Overview | What is SD WAN | Benefits of SD WAN
Ashutosh Kaushik
 
POWER POINT PRESENTATION ON DATA CENTER
vivekprajapatiankur
 
Data Center Networks
Sayed Chhattan Shah
 
SDWAN vs MPLS: What Enterprises need?
Haris Chughtai
 
Software Defined Network - SDN
Venkata Naga Ravi
 
Datacenter overview
Heather Brotherton
 
DataCenter:: Infrastructure Presentation
Muhammad Asad Rashid
 
SD WAN
Bri Molina
 
Firewall presentation
TayabaZahid
 
Introduction to Data Center Network Architecture
Ankita Mahajan
 
Router Full Concept pdf
Vignesh kumar
 
Network Address Translation (NAT)
Joud Khattab
 
ClearPass Overview
JoAnna Cheshire
 
Customer Presentation - Aruba Wi-Fi Overview (1).PPTX
ssuser5824cf
 
Real-world 802.1X Deployment Challenges
Aruba, a Hewlett Packard Enterprise company
 
Data Centre Design Guideline and Standards
Bangladesh Network Operators Group
 
Data Center Infrastructure Management Powerpoint Presentation Slides
SlideTeam
 
Private Cloud Architecture
Derek Keats
 
Overview of Major Aruba Switching Features incl. Smart Rate for Multi-Gig Ports
Aruba, a Hewlett Packard Enterprise company
 
Fortigate Training
NCS Computech Ltd.
 
Ad

Viewers also liked (20)

PPTX
Data center network architectures v1.3
Jeong, Wookjae
 
PDF
Dell Data Center Networking Overview
Dell World
 
PDF
Designing Secure Cisco Data Centers
Cisco Russia
 
PPT
A Scalable, Commodity Data Center Network Architecture
Gunawan Jusuf
 
PDF
Security and Virtualization in the Data Center
Cisco Canada
 
PPT
Tia 942 Data Center Standards
Sri Chalasani
 
PDF
Data Center Security
Cisco Canada
 
PDF
Data Center Virtualization @ Cisco
vmug
 
PPTX
FATTREE: A scalable Commodity Data Center Network Architecture
Ankita Mahajan
 
PDF
Modern Data Center Network Architecture - The house that Clos built
Cumulus Networks
 
PDF
Data center Building & General Specification
Ali Mirfallah
 
PDF
Reference Architecture-Validated & Tested Approach to Define Network Design
DataWorks Summit
 
PDF
Data Center Architecture Trends
Panduit
 
PPTX
Introduction to Cloud Data Center and Network Issues
Jason TC HOU (侯宗成)
 
PPT
Simplifying Data Center Design/ Build
Schneider Electric
 
DOC
Data Center Proposal (System Network Administration)
Muhammad Ahad
 
PDF
Enterprise data center design and methodology
Carlos León Araujo
 
PPTX
Data Center Trends 2014
Belden Inc
 
PDF
Murphy Data Center Services Infographic
Gary Woodcock
 
PPT
Genesys System - 8dec2010
Agora Group
 
Data center network architectures v1.3
Jeong, Wookjae
 
Dell Data Center Networking Overview
Dell World
 
Designing Secure Cisco Data Centers
Cisco Russia
 
A Scalable, Commodity Data Center Network Architecture
Gunawan Jusuf
 
Security and Virtualization in the Data Center
Cisco Canada
 
Tia 942 Data Center Standards
Sri Chalasani
 
Data Center Security
Cisco Canada
 
Data Center Virtualization @ Cisco
vmug
 
FATTREE: A scalable Commodity Data Center Network Architecture
Ankita Mahajan
 
Modern Data Center Network Architecture - The house that Clos built
Cumulus Networks
 
Data center Building & General Specification
Ali Mirfallah
 
Reference Architecture-Validated & Tested Approach to Define Network Design
DataWorks Summit
 
Data Center Architecture Trends
Panduit
 
Introduction to Cloud Data Center and Network Issues
Jason TC HOU (侯宗成)
 
Simplifying Data Center Design/ Build
Schneider Electric
 
Data Center Proposal (System Network Administration)
Muhammad Ahad
 
Enterprise data center design and methodology
Carlos León Araujo
 
Data Center Trends 2014
Belden Inc
 
Murphy Data Center Services Infographic
Gary Woodcock
 
Genesys System - 8dec2010
Agora Group
 
Ad

Similar to Data Center Network Topologies (20)

PDF
Graphs are at the Heart of the Cloud
Alejandro Erickson
 
PPTX
Lecture notes - Data Centers________.pptx
SandeepGupta229023
 
PPTX
Advanced Design and Optimization of Data Center Interconnection Networks.pptx
Service Solutions Pvt. Ltd. (SSL)
 
PPTX
LinkedIn's Approach to Programmable Data Center
Shawn Zandi
 
PPTX
DC Moving I migracion a otro datacenterd
ElvisJessVillalva
 
PDF
Dcnintroduction 141010054657-conversion-gate01
yibeltal yideg
 
PPT
Theo's slides
webhostingguy
 
PPT
Theo's slides
webhostingguy
 
PPTX
Cloud interconnection networks basic .pptx
RahulBhole12
 
PPTX
Chapter 1 - Network Topologies.pptx
NaeemAkhtar403523
 
PPTX
20-datacenter-measurements.pptx
Steve491226
 
PPTX
20-datacenterdasdasdas-measurements.pptx
dusanheliant
 
PDF
Ccnp presentation day 4 sd-access vs traditional network architecture
SagarR24
 
PDF
Optical Switching in the Datacenter
Kostas Katrinis
 
PPTX
Designing network topology.pptx
KISHOYIANKISH
 
PDF
Presentation on Data Center Use-Case & Trends
Amod Dani
 
PDF
5G-USA-Telemetry
snrism
 
DOCX
original
sujal Khatiwada
 
PPT
Fernando Loureiro Presentation / CloudViews.Org - Cloud Computing Conference ...
EuroCloud
 
PDF
Large Scale Data center Solution Guide: eBGP based design
Dhiman Chowdhury
 
Graphs are at the Heart of the Cloud
Alejandro Erickson
 
Lecture notes - Data Centers________.pptx
SandeepGupta229023
 
Advanced Design and Optimization of Data Center Interconnection Networks.pptx
Service Solutions Pvt. Ltd. (SSL)
 
LinkedIn's Approach to Programmable Data Center
Shawn Zandi
 
DC Moving I migracion a otro datacenterd
ElvisJessVillalva
 
Dcnintroduction 141010054657-conversion-gate01
yibeltal yideg
 
Theo's slides
webhostingguy
 
Theo's slides
webhostingguy
 
Cloud interconnection networks basic .pptx
RahulBhole12
 
Chapter 1 - Network Topologies.pptx
NaeemAkhtar403523
 
20-datacenter-measurements.pptx
Steve491226
 
20-datacenterdasdasdas-measurements.pptx
dusanheliant
 
Ccnp presentation day 4 sd-access vs traditional network architecture
SagarR24
 
Optical Switching in the Datacenter
Kostas Katrinis
 
Designing network topology.pptx
KISHOYIANKISH
 
Presentation on Data Center Use-Case & Trends
Amod Dani
 
5G-USA-Telemetry
snrism
 
original
sujal Khatiwada
 
Fernando Loureiro Presentation / CloudViews.Org - Cloud Computing Conference ...
EuroCloud
 
Large Scale Data center Solution Guide: eBGP based design
Dhiman Chowdhury
 

More from rjain51 (18)

PDF
Internet of Things: Challenges and Issues
rjain51
 
PDF
SDN and NFV: Facts, Extensions, and Carrier Opportunities
rjain51
 
PDF
Introduction to Internet of Things
rjain51
 
PDF
Introduction to Network Function Virtualization (NFV)
rjain51
 
PDF
Introduction to Software Defined Networking (SDN)
rjain51
 
PDF
OpenFlow Controllers and Tools
rjain51
 
PDF
Introduction to OpenFlow
rjain51
 
PDF
Network Virtualization in Cloud Data Centers
rjain51
 
PDF
LAN Extension and Network Virtualization for Cloud Computing using Layer 3 Pr...
rjain51
 
PDF
Networking Issues For Big Data
rjain51
 
PDF
Big Data Fundamentals
rjain51
 
PDF
Data Center Networks:Virtual Bridging
rjain51
 
PDF
Application Delivery Networking
rjain51
 
PDF
Carrier Ethernet
rjain51
 
PDF
Storage Virtualization
rjain51
 
PDF
Server Virtualization
rjain51
 
PDF
Data Center Ethernet
rjain51
 
PDF
Networking Protocols for Internet of Things
rjain51
 
Internet of Things: Challenges and Issues
rjain51
 
SDN and NFV: Facts, Extensions, and Carrier Opportunities
rjain51
 
Introduction to Internet of Things
rjain51
 
Introduction to Network Function Virtualization (NFV)
rjain51
 
Introduction to Software Defined Networking (SDN)
rjain51
 
OpenFlow Controllers and Tools
rjain51
 
Introduction to OpenFlow
rjain51
 
Network Virtualization in Cloud Data Centers
rjain51
 
LAN Extension and Network Virtualization for Cloud Computing using Layer 3 Pr...
rjain51
 
Networking Issues For Big Data
rjain51
 
Big Data Fundamentals
rjain51
 
Data Center Networks:Virtual Bridging
rjain51
 
Application Delivery Networking
rjain51
 
Carrier Ethernet
rjain51
 
Storage Virtualization
rjain51
 
Server Virtualization
rjain51
 
Data Center Ethernet
rjain51
 
Networking Protocols for Internet of Things
rjain51
 

Recently uploaded (20)

PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
PDF
Trying to figure out MCP by actually building an app from scratch with open s...
Julien SIMON
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PDF
Brief History of Internet - Early Days of Internet
sutharharshit158
 
PDF
The Evolution of KM Roles (Presented at Knowledge Summit Dublin 2025)
Enterprise Knowledge
 
PDF
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PDF
Event Presentation Google Cloud Next Extended 2025
minhtrietgect
 
PPTX
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
PDF
REPORT: Heating appliances market in Poland 2024
SPIUG
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PPTX
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
PDF
AI-Cloud-Business-Management-Platforms-The-Key-to-Efficiency-Growth.pdf
Artjoker Software Development Company
 
PDF
How-Cloud-Computing-Impacts-Businesses-in-2025-and-Beyond.pdf
Artjoker Software Development Company
 
PDF
Doc9.....................................
SofiaCollazos
 
PDF
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
PDF
Advances in Ultra High Voltage (UHV) Transmission and Distribution Systems.pdf
Nabajyoti Banik
 
PPTX
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
Trying to figure out MCP by actually building an app from scratch with open s...
Julien SIMON
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
Brief History of Internet - Early Days of Internet
sutharharshit158
 
The Evolution of KM Roles (Presented at Knowledge Summit Dublin 2025)
Enterprise Knowledge
 
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
Event Presentation Google Cloud Next Extended 2025
minhtrietgect
 
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
REPORT: Heating appliances market in Poland 2024
SPIUG
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
cloud computing vai.pptx for the project
vaibhavdobariyal79
 
AI-Cloud-Business-Management-Platforms-The-Key-to-Efficiency-Growth.pdf
Artjoker Software Development Company
 
How-Cloud-Computing-Impacts-Businesses-in-2025-and-Beyond.pdf
Artjoker Software Development Company
 
Doc9.....................................
SofiaCollazos
 
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
Advances in Ultra High Voltage (UHV) Transmission and Distribution Systems.pdf
Nabajyoti Banik
 
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 

Data Center Network Topologies

  • 1. Data Center Network Topologies . Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 [email protected] These slides and audio/video recordings of this class lecture are at: http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-1 ©2013 Raj Jain
  • 2. Overview 1. 2. 3. 4. 5. Data Center Physical Layout Data Center Network Topologies ToR vs. EoR Data Center Networking Issues Data Center Networking Requirements Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-2 ©2013 Raj Jain
  • 3. Google’s Data Center Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-3 ©2013 Raj Jain
  • 5. Modular Data Centers      Small: < 1 MW, 4 racks per unit Medium: 1-4 MW, 10 racks per unit Large: > 4 MW, 20 racks per unit Built-in cooling, high PUE (power usage effectiveness) 1.02 PUE = Power In/Power Used Rapid deployment Ref: http://www.sgi.com/products/data_center/ice_cube_air/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-5 ©2013 Raj Jain
  • 6. Containerized Data Center     Ready to Use. Connect to water and power supply and go. Built in cooling. Easy to scale.  Data Center trailer parks. Suitable for disaster recovery, e.g., flood, earthquake Offered by Cisco, IBM, SGI, Sun/ORACLE,… Ref: Datacenter Infrastructure – mobile Data Center from Emerson Network Power , http://en.m-info.ua/180-container-data-center/755-datacenter-infrastructure-mobile-data-center-from-emerson-network-power Ref: http://www.datacenterknowledge.com/archives/2010/05/31/iij-will-offer-commercial-container-facility/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-6
  • 7. Unstructured Cabling Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-7 ©2013 Raj Jain
  • 8. Structured Cabling Source: http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-8 ©2013 Raj Jain
  • 9. Data Center Equipment Cabinets Three Layers: Bottom: Signal, Middle: Power, Top: Fiber Minimize patching between cabinets and racks Cabling under raised floors provides better appearance and cooling Ref: Ref: C. DiMinico, "Telecommunications Infrastructure Standard for Data Centers," IEEE 802.3 HSSG Meeting, Nov. 2006, http://www.ieee802.org/3/hssg/public/nov06/diminico_01_1106.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-9
  • 10. Data Center Physical Layout Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-10 ©2013 Raj Jain
  • 11. ANSI/TIA-942-2005 Standard     Main Distribution Area (MDA) Horizontal Distribution Area (HDA) Equipment Distribution Area (EDA) Zone Distribution Area (ZDA) Source: Santana 2014 Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-11 ©2013 Raj Jain
  • 12. ANSI/TIA-942-2005 Standard Computer Room: Main servers  Entrance Room: Data Center to external cabling  Cross-Connect: Enables termination of cables  Main Distribution Area (MDA): Main cross connect. Central Point of Structured Cabling. Core network devices  Horizontal Distribution Area (HDA): Connections to active equipment.  Equipment Distribution Area (EDA): Active Servers+Switches. Alternate hot and cold aisle.  Zone Distribution Area (ZDA): Optionally between HDA and EDA. ZDA allows easy  Backbone Cabling: Connections between MDA, HDA, and Entrance room http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain  3-12
  • 13. Zone Distribution Area  High-fiber count cables connect ZDA to MDA or HDA. Low-fiber count cables connect ZDA to EDA as needed. Ref: Jennifer Cline, “Zone Distribution in the data center,” http://www.graybar.com/documents/zone-distribution-in-the-data-center.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-13 ©2013 Raj Jain
  • 14. Data Center Network Topologies  Core, Aggregation, Access Ref: G. Santana, “Data Center Virtualization Fundamentals,” Cisco Press, 2014, ISBN:1587143240 http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-14 ©2013 Raj Jain
  • 15. Data Center Networks        20-40 servers per rack Each server connected to 2 access switches with 1 Gbps (10 Gbps becoming common) Access switches connect to 2 aggregation switches Aggregation switches connect to 2 core routers Core routers connect to edge routers Aggregation layer is the transition point between L2-switched access layer and l3-routed core layer Low Latency: In high-frequency trading market, a few microseconds make a big difference.  Cut-through switching and low-latency specifications. Ref: A. Greenberg, "VL2: A Scalable and Flexible Data Center Network," CACM, Vol. 54, NO. 3, March 2011, pp. 95-104, http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf. http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-15
  • 16. Data Center Networks (Cont)       Core routers manage traffic between aggregation routers and in/out of data center All switches below each pair of aggregation switches form a single layer-2 domain Each Layer 2 domain typically limited to a few hundred servers to limit broadcast Most traffic is internal to the data center. Network is the bottleneck. Uplinks utilization of 80% is common. Most of the flows are small. Mode = 100 MB. DFS uses 100 MB chunks. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-16 ©2013 Raj Jain
  • 17. Switch Locations Smaller cable between servers and switches Network team has to manage switches on all racks All network switches in one rack Source: Santana 2014 Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-17 ©2013 Raj Jain
  • 18. ToR vs EoR  ToR:  Easier cabling    If rack is not fully populated  unused ToR ports If rack traffic demand is high, difficult to add more ports Upgrading (1G to 10G) requires complete Rack upgrade   EoR:  Longer cables  Severs can be place in any rack  Ports can easily added, upgraded Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-18 ©2013 Raj Jain
  • 19. Hierarchical Network Design     All servers require application delivery services for security (VPN, Intrusion detection, firewall), performance (load balancer), networking (DNS, DHCP, NTP, FTP, RADIUS), Database services (SQL) ADCs are located between the aggregation and core routers and are shared by all servers Stateful devices (firewalls) on Aggregation layer Stateful= State of TCP connection Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-19 Source: Santana 2014 ©2013 Raj Jain
  • 20. Access Aggregation Connections 1. 2. 3. 4. Looped Triangle: Most common. Spanning Tree Protocol (STP) blocks links. Paid but not used. Looped Square: Oversubscription doubles if failure. Loop-Free U: No L2 communication between aggregation switches if any switch links fail Loop-Free Inverted U: Black-holes on some failures Washington University in St. Louis Source: Santana 2014 http://www.cse.wustl.edu/~jain/cse570-13/ 3-20 ©2013 Raj Jain
  • 21. Data Center Networking Issues     Higher layers oversubscribed:  Other servers in the same rack 1:1  Uplinks from ToR: 1:2 to 1:20 (e.g., 32x10Gb down, 8X10Gb up  4:1 oversubscription)  Core Routers: 1:240  Generally keep services in one tree  Can't arbitrarily move servers Moving across Subnets is painful  Requires reconfiguration of IP addresses and VLAN trunks Service trample on each-other. Overuse by one service affects others Poor reliability. One access switch failure doubles the load on the other. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-21 ©2013 Raj Jain
  • 22. Data Center Networking Issues (Cont)   Under-utilization. Even when multiple paths exist only one is used. ECMP (Equal Cost Multipath) is used by routers to spread traffic to next hops using a hash function. However, only 2 paths exist. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-22 ©2013 Raj Jain
  • 23. DCN Requirements  Needs to be Scalable, Secure, Shared, Standardized, and Simplified (5 S's)  Converged Infrastructure: Servers, storage, and network have to work together Workload Mobility: Large L2 domains required for VM mobility East-West Traffic: Significant server-to-server traffic as compared to server to user. One Facebook request required 88 cache looks, 35 database lookups, 392 backend RPC calls. Internet traffic 935X the http request/response [Farrington] Storage traffic on Ethernet: Congestion management on Ethernet    Ref: A. Kindlness, "The Forester Wave: Data Center Networking Hardware," Jan 23, 2013, http://ca.westcon.com/documents/46488/forrester_wave_data_cetner_networking_hw_q1_2013.pdf Ref: N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-23
  • 24. 4-Post Architecture at Facebook     Each rack switch (RSW) has up to 48 10G downlinks and 4-8 10G uplinks (10:1 oversubscription) to cluster switch (CSW) Each CSW has 4 40G uplinks – one to each of the 4 FatCat (FC) aggregation switches (4:1 oversubscription) 4 CSW’s are connected in a 10G×8 protection ring 4FC’s are connected in a 10G×16 protection ring No routers at FC. One CSW failure reduces intra-cluster capacity to 75%. Ref: N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis ©2013 Raj Jain 3-24
  • 25. Clos Networks        Multi-stage circuit switching network proposed by Charles Clos in 1953 for telephone switching systems Allows forming a large switch from smaller switches The number of cross-points is reduced  Lower cost (then) 3-Stage Clos(n, m, r): ingress (r n×m), middle (m r×r), egress (r m×n) Strict-sense non-blocking if m > 2n-1. Existing calls unaffected. Rearrangeably non-blocking if m > n Can have any odd number of stages, e.g., 5 Folded: Merge input and output in to one switch = Fat-tree n×m n×m r=2, n=4, m=3 r×r r×r r×r Ref: http://en.wikipedia.org/wiki/Clos_network Washington University in St. Louis r×r m×n r×r m×n r×r n+m port switches r-port switches http://www.cse.wustl.edu/~jain/cse570-13/ 3-25 ©2013 Raj Jain
  • 26. Fat-Tree DCN Example Core 9 Edge Servers     … … … … 6 identical 36-port switches. All ports 1 Gbps. 72 Servers. Each edge switch connects to 18 servers. 9 Uplinks to first core switch. Other 9 links to 2nd core switch. Throughput between any two servers = 1 Gbps using ECMP Identical bandwidth at any bisection. Negative: Cabling complexity Ref: Teach yourself Fat-Tree Design in 60 minutes, http://clusterdesign.org/fat-trees/ http://www.cse.wustl.edu/~jain/cse570-13/ Washington University in St. Louis 3-26 ©2013 Raj Jain
  • 27. Summary 1. 2. 3. 4. 5. Modular data centers can be used for easy assembly and scaling Three tiers: Access, Aggregation, Core Application delivery controllers between Aggregation and core Need large L2 domains Fat-tree topology is sometimes used to improve performance and reliability Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-27 ©2013 Raj Jain
  • 28. Homework 3  Draw a 3-stage clos(4,5,3) topology and its folded version. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-28 ©2013 Raj Jain
  • 29. Acronyms ADC ANSI BPE CSW DCBX DCN DFS DHCP DNS ECMP EDA EoR Application Delivery Controller American National Standards Institute Business Process Engineering Core Switch Data Center Bridging eXtension Data Center Network Distributed File System Dynamic Host Control Protocol Domain Name System Equal Cost Multipath Equipment Distribution Area End of Row Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-29 ©2013 Raj Jain
  • 30. Acronyms (Cont) ETS Enhanced Transmission Selection EVB Edge Virtual Bridge FC Fibre Channel FSW Fabric switch FTP File Transfer Protocol HDA Horizontal Distribution Area LACP Link Aggregation Control Protocol LAG Link Aggregation LLDP Link Layer Discovery Protocol MAC Media Access Control MDA Main Distribution Area MW Mega-Watt NTP Network Time Protocol Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-30 ©2013 Raj Jain
  • 31. Acronyms (Cont) NVGRE Network Virtualization using Generic Routing Encapsulation PFC Priority Flow Control PUE Power Usage Effectiveness RADIUS Remote Authentication Dial-In User Service RPC Remote Procedue Call RSW Rack switch SQL Structured Query Language SSW Spine Switches STP Spanning Tree Protocol TIA Telecommunications Industry Association ToR Top of Rack TRILL Transparent Interconnection of Lots of Link VLAN Virtual Local Area Network VM Virtual Machine VPN Virtual Private Network Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-31 ©2013 Raj Jain
  • 32. Acronyms (Cont) VRF Virtual Routing and Forwarding VXLANVirtual Extensible Local Area Network ZDA Zone Distribution Area Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-32 ©2013 Raj Jain
  • 33. Reading List        http://webodysseum.com/technologyscience/visit-the-googles-data-centers/ http://www.sgi.com/products/data_center/ice_cube_air/ Datacenter Infrastructure - mobile Data Center from Emerson Network Power, http://www.datacenterknowledge.com/archives/2010/05/31/iij-willoffer-commercial-container-facility/ C. DiMinico, "Telecommunications Infrastructure Standard for Data Centers," IEEE 802.3 HSSG Meeting, Nov. 2006, http://www.ieee802.org/3/hssg/public/nov06/diminico_01_1106.pdf Jennifer Cline, “Zone Distribution in the data center,” http://www.graybar.com/documents/zone-distribution-in-the-datacenter.pdf G. Santana, “Data Center Virtualization Fundamentals,” Cisco Press, 2014, ISBN:1587143240 (Safari book) A. Greenberg, "VL2: A Scalable and Flexible Data Center Network," CACM, Vol. 54, NO. 3, March 2011, pp. 95-104, http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-33 ©2013 Raj Jain
  • 34. Reading List (Cont)      A. Kindlness, "The Forester Wave: Data Center Networking Hardware," Jan 23, 2013, http://ca.westcon.com/documents/46488/forrester_wave_data_cetner_netwo rking_hw_q1_2013.pdf N. Farringon and A. Andreyev, “Facebook’s Data Center Network Architecture,” 2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf http://en.wikipedia.org/wiki/Clos_network Teach yourself Fat-Tree Design in 60 minutes, http://clusterdesign.org/fattrees/ M. Al-Fares, et al, "A scalable, commodity data center network architecture," ACM SIGCOMM, 2008. Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-34 ©2013 Raj Jain
  • 35. Wikipedia Links          http://en.wikipedia.org/wiki/Modular_data_center http://en.wikipedia.org/wiki/Data_center http://en.wikipedia.org/wiki/Structured_cabling http://en.wikipedia.org/wiki/Cable_management http://en.wikipedia.org/wiki/Raised_floor http://en.wikipedia.org/wiki/Data_center_environmental_contr ol http://en.wikipedia.org/wiki/Fat_tree http://en.wikipedia.org/wiki/Hierarchical_internetworking_mod el http://en.wikipedia.org/wiki/Clos_network Washington University in St. Louis http://www.cse.wustl.edu/~jain/cse570-13/ 3-35 ©2013 Raj Jain