FCOE Convergence at the Access Layer with Juniper Networks QFX3500 Switch
1 like1,043 views
The document discusses the FCoE convergence capabilities at the access layer using the Juniper Networks QFX3500 switch, which addresses challenges of server and access-layer I/O convergence. It outlines various operational modes for convergence products and emphasizes the importance of developing standards for efficient data center networking. The QFX3500 is highlighted as a cutting-edge solution that supports high bandwidth and multiple connectivity options, enabling organizations to leverage their existing network investments while preparing for future developments.
FCOE Convergence at the Access Layer with Juniper Networks QFX3500 Switch
- 1. WHITE PAPER FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch First Top-of-Rack Switch Built to Solve All the Challenges Posed by Access-Layer Convergence Copyright © 2011, Juniper Networks, Inc. 1
- 2. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Table of Contents Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Access-Layer Convergence Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Option 1: FCoE Transit Switch (DCB Switch with FIP Snooping) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 FCoE Servers with CNA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Option 2: FCoE-FC Gateway (Using NPIV Proxy). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Option 3: FCoE-FC Switch (Full FCF) (Not Recommended). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Deployment Models Available Today. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Rack-Mount Servers and Top-of-Rack FCoE-FC Gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Blade Servers with Pass-Through Modules and Top-of-Rack FCoE-FC Gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Blade Servers with Embedded DCB Switch and Top-of-Rack FCoE-FC Gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Blade Servers with Embedded FCoE-FC Gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Servers Connected Through FCoE Transit Switch to an FCoE-Enabled Fibre Channel SAN Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . 10 The Standards that Allow for Server I/O and Access-Layer Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Enhancements to Ethernet for Converged Data Center Networks—DCB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Enhancements to Fibre Channel for Converged Data Center Networks—FCoE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Future Direction for FCoE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 A Brief Note on iSCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 About Juniper Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Table of Figures Figure 1: The phases of convergence, from separate networks, to access layer convergence, to the fully converged network.. . . 4 Figure 2: Operation FCoE transit switch vs. FCoE-FC gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 3: Operation of an FCoE transit switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Figure 4: FCoE servers with CNA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Figure 5: Rack-mount servers and top-of-rack FCoE-FC gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 6: Blade servers with pass-through modules and top-of-rack FCoE-FC gateway.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 7: Blade servers with embedded DCB switch and top-of-rack FCoE-FC gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 8: Servers connected to FCoE transit switch through to an FCoE-enabled FC SAN fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 9: PFC ETS and QCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 Copyright © 2011, Juniper Networks, Inc.
- 3. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Executive Summary In 2011, customers will finally be able to invest in convergence-enabling equipment and begin reaping the benefits of convergence in their data centers. With the first wave of standards now complete—both the IEEE Data Center Bridging (DCB) enhancements to Ethernet and the InterNational Committee for Information Technology Standards (INCITS) T11 FC-BB-5 standard for Fibre Channel over Ethernet (FCoE), enterprises can benefit from server- and access-layer I/O convergence while continuing to leverage their investment in their existing aggregation, core LAN, and Fibre Channel (FC) backbones. So why the focus on server and access-layer I/O convergence? Simply put, the industry recognizes that the first wave of standards does not meet the needs of full convergence and so it is working on a second wave of standards—including FC- BB-6—as well as various forms of fabric technology to better address the challenges of full convergence. The new standards are designed to provide lower cost convergence strategies for the smaller enterprise, and to address the scaling issues that come about from convergence in general as well as increased data center scale. As a result, 2011 is the year to focus on the benefits to be gained from converging the access layer while laying a foundation for the future. Juniper Networks® QFX3500 Switch is the first top-of-rack switch built to solve all of the challenges posed by access-layer convergence. It works for both rack-mount and blade servers, and for organizations with combined or separate LAN and storage area network (SAN) teams. It is also the first product to leverage a new generation of ASIC technologies. It offers 1.28 terabits per second (Tbps) of bandwidth implemented with a single ultra-low latency ASIC and soft-programmable ports capable of gigabit Ethernet (GbE), 10GbE, 40GbE, and 2/4/8 Gbps FC, supported through small form-factor pluggable transceiver (SFP+) GbE copper, 10GbE copper and optical digital to analog converter (DAC), and quad small form-factor pluggable (QSFP) dense optical connectivity. Copyright © 2011, Juniper Networks, Inc. 3
- 4. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch SAN A SAN B Phase 1: Separate Networks SAN A SAN B Phase 2: Access Layer Convergence Phase 3: Full Convergence Figure 1: The phases of convergence, from separate networks, to access layer convergence, to the fully converged network. 4 Copyright © 2011, Juniper Networks, Inc.
- 5. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Introduction The network is the critical enabler of all services delivered from the data center. A simple, streamlined, and scalable data center network fabric can deliver greater efficiency and productivity, as well as lower operating costs. Such a network also allows the data center to support much higher levels of business agility and not become a bottleneck that hinders a company from releasing new products or services. To allow businesses to make sound investment decisions, this white paper will look at the following areas to fully clarify the most interesting options for convergence in 2011: 1. Review the different types of convergence-capable products that are available on the market based upon the current standards and consider the capabilities of those products 2. Consider the deployment scenarios for those products 3. Look forward to some of the new product and solution capabilities expected over the next couple of years Access-Layer Convergence Modes When buying a convergence platform, it is possible to deploy products based on three very different modes of operation. Products on the market today may be capable of one or more of these modes depending on hardware and software configuration and license enablement. • FCoE transit switch—DCB switch with FCoE Initialization Protocol (FIP) snooping • FCoE-FC gateway—using N_Port ID Virtualization (NPIV) proxy • FCoE-FC switch—full Fibre Channel Forwarder (FCF) capability In principle, these systems can be used in multiple places within a deployment. However, for the purpose of this document and based on the most likely deployments in 2011, only the server access-layer convergence model will be covered. FCOE Transit Switch vs. FCOE-FC Gateway FC/FCoE Switch FC Switch VF_Port VF_Port VF_Port DCB DCB F_Port F_Port Port Port DCB DCB N_Port N_Port Port Port FCoE Transit Switch NPIV Proxy FIP Snooping FIP FIP FIP VF_Port VF_Port VF_Port ACL ACL ACL DCB DCB DCB DCB DCB DCB Port Port Port Port Port Port VN_Port VN_Port VN_Port VN_Port VN_Port VN_Port FCoE servers with CNA FCoE servers with CNA Figure 2: Operation FCoE transit switch vs. FCoE-FC gateway Copyright © 2011, Juniper Networks, Inc. 5
- 6. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Option 1: FCoE Transit Switch (DCB Switch with FIP Snooping) In this model, the SAN team enables their backbone SAN fabric for FCoE, while the network team deploys a top-of-rack DCB switch with FIP snooping. Servers are deployed with Converged Network Adapters (CNAs), and blade servers are deployed with pass-through modules or embedded DCB switches. These are connected to the top-of-rack switch, which then has Ethernet connectivity to the LAN aggregation layer and Ethernet connectivity to the FCoE ports of the SAN backbone. A common question at this point is whether a DCB switch with no Fibre Channel stack can indeed be a viable part of a converged deployment and, in particular, whether such a switch gives not just the necessary security but also the performance and manageability required in a storage network deployment. Since this is, at one level, just a Layer 2 switch, this solution ensures that the switch in each server rack is not consuming an FC domain ID. Fibre Channel networks have a scale restriction that limits them to just a couple of tens of switches. As convergence and 10GbE forces a move towards top-of-rack switches, any solution deployed must ensure that convergence does not cause an FC SAN scaling problem. FCoE FCoE Enabled SAN LAG LAG FCoE FCoE Figure 3: Operation of an FCoE transit switch FCoE Servers with CNA A rich implementation of an FCoE transit switch will provide strong management and monitoring of the traffic separation, allowing the SAN team to monitor FCoE traffic throughput. Specifically, a fully manageable DCB switch will allow the user to monitor traffic on a per user priority and per priority group basis and not just per port. FIP snooping as defined in the FCoE standard provides perimeter protection, ensuring that the presence of an Ethernet layer in no way impacts existing SAN security. The SAN backbone can be simply FCoE-enabled with either FCoE blades within chassis-based systems or FCoE-FC gateways connected to the edge of the SAN backbone. In addition, the traditional Fibre Channel Security Profile (FC-SP) mechanisms work seamlessly through FCoE, allowing CNA-to-FCF authentication to be used through the DCB switch. Perhaps less obviously, FIP snooping also means that the switch has a very clear view of each and every FCoE session that is running through it, both in terms of the path, which is derived from the source and destination media access control (MAC) frames of the virtual Fibre Channel ports, as well as the actual status of the virtual FC connection, which is monitored by snooping the FIP keepalive. 6 Copyright © 2011, Juniper Networks, Inc.
- 7. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Just as with any Ethernet deployment, the switch can use link aggregation group (LAG) to balance the Ethernet packets (including FCoE) across multiple links. As with any FC switch, this load balancing can include the OxID (Fibre Channel exchange ID) in order to carry out the Fibre Channel best practice of exchange-based load balancing. Finally, the FCoE protocol includes FCoE load-balancing capabilities to ensure that the FCoE servers are evenly and appropriately distributed across the multiple FCoE FC fabric connections. FCoE transit switches have several advantages: • Low-cost top-of-rack DCB switch • Rich monitoring of FCoE traffic at top of rack (QFX3500 Switch) • FCoE enablement of SAN backbone (FCoE blades or FCoE-FC gateway) managed by the SAN team for clean management separation • Load balancing carried out between CNAs and FCoE ports of the SAN fabric as well as point-to-point throughout the Ethernet infrastructure • Comprehensive security maintained through FIP snooping and FC-SP • No heterogeneous support issues, as top of rack is L2 connectivity only FC FC SAN LAG LAG FC FCoE Figure 4: FCoE servers with CNA Option 2: FCoE-FC Gateway (Using NPIV Proxy) In this model, the SAN and Ethernet teams agree jointly to deploy an FCoE-FC top-of-rack gateway. From a cabling perspective, the deployment is identical to Option 1, with the most visible difference being that the cable between the top of rack and the SAN backbone is now carrying native Fibre Channel traffic rather than FCoE traffic. As with Option 1, this solution ensures that the switch in each server rack is not consuming an FC domain ID. In this case, however, unlike Option 1, a much richer level of Fibre Channel functionality has been enabled within the switch. The FCoE- FC gateway uses NPIV technology so that it presents to the servers as an FCoE-enabled Fibre Channel switch, and presents to the SAN backbone as a group of FC servers. It then simply proxies sessions from one domain to the other with intelligent load-balancing and automated failover capability across the Fibre Channel links to the fabric. Copyright © 2011, Juniper Networks, Inc. 7
- 8. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch FCoE-FC gateways have several advantages: • Clean separation of management through role-based access control (QFX3500 Switch) • No need for FCoE enablement of the SAN backbone • Fine-grained FCoE session-based load balancing (at the virtual machine level for NPIV-enabled hypervisors—QFX3500 Switch) and full Ethernet LAG with exchange-based load balancing on the Ethernet-facing connectivity • No heterogeneous support issues, as the FCoE-FC gateway presents to the SAN fabric as a Fibre Channel-enabled server (N_Port to F_Port) • Available post deployment as a license upgrade and fungible port reconfiguration with no additional hardware (QFX3500 Switch) • Support for an upstream DCB switch such as an embedded switch in blade server shelf (QFX3500 Switch), as well as direct CNA connectivity or connectivity via blade server pass-through modules Option 3: FCoE-FC Switch (Full FCF) (Not Recommended) For deployments of any size, there is no value to local switching, as any rack is either pure server or pure storage. In addition, although the SAN standards limit deployments to 239 switches, the practical supported limits are typically within the 16 to 32 range (in reality, most deployments are kept well below these limits). As such, this option has limited value in production data centers. For very small configurations where a single switch needs to connect to both servers and storage, Juniper believes that Internet Small Computer System Interface (iSCSI) is the best approach in 2011, while the FC-BB-6 VN2VN model (see “Future Direction for FCoE” section later in this white paper) will be the preferred FCoE end-to-end model in 2012. Deployment Models Available Today As previously noted, this paper focuses on deployments that apply for server access layer convergence. As such, it is assumed that this access layer is in turn connecting both to some form of Ethernet aggregation/core layer on one side and a Fibre Channel backbone on the other. The term “Fibre Channel backbone” implies a traditional FC SAN of some form which has attached to the FC disk and tape as well as most likely existing FC servers. By leveraging either an FCoE transit switch or an FCoE-FC gateway, whether separately or together, there are a number of deployment options for supporting both rack-mount servers and blade servers. Each approach has its merits, and organizations may want to use different approaches, depending on their requirements. In terms of physical deployment in most data centers, the Ethernet aggregation and core, the FC backbone, and the FC disk and tape are likely to be colocated in some centralized location within the data center with the server racks. From a cabling perspective, this means that the same physical cable infrastructure can easily support any of the deployment models discussed below. Rack-Mount Servers and Top-of-Rack FCoE-FC Gateway This deployment model is perhaps the most recognized and best understood. The QFX3500 Switch fully supports this model and, unlike other products, the QFX3500 enables this mode through a single license that allows up to 12 of its 48 SFP+ ports to be configured for 2/4/8 Gbps FC instead of 10GbE. 8 Copyright © 2011, Juniper Networks, Inc.
- 9. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch FC Ethernet SAN Network Ethernet FC Figure 5: Rack-mount servers and top-of-rack FCoE-FC gateway Blade Servers with Pass-Through Modules and Top-of-Rack FCoE-FC Gateway This model is similar to the previous rack-mount servers and top-of-rack FCoE-FC gateway model. The challenge with this model is the complex cabling that accompanies pass-through modules. Using pass-through has the benefit of removing an entire layer from the network topology, thereby simplifying the data center, ensuring a single network operating system at all layers, and allowing the edge of the network to leverage the richer functionality available with the feature-rich ASICs used at top of rack. The use of modern pass-through modules and well constructed cabling solutions provide all the cable simplicity benefits of an embedded blade switch with none of the limitations. TOR FCoE-FC Gateway Pass-Through Module Figure 6: Blade servers with pass-through modules and top-of-rack FCoE-FC gateway. Copyright © 2011, Juniper Networks, Inc. 9
- 10. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Blade Servers with Embedded DCB Switch and Top-of-Rack FCoE-FC Gateway To support this deployment model, it is necessary to ensure that both the CNAs and the FCoE-FC gateway have particularly feature-rich implementations of the full FC-BB-5 standard in order to support many-to-many L2 visibility for fan-in load balancing and high availability. The QFX3500 Switch is the first fully FC-BB-5-enabled gateway capable of easily supporting upstream DCB switches, including third-party embedded blade shelf switches. Juniper strongly recommends using such switches only if they have implemented FIP snooping for perimeter detection, and they have fully standards-based, feature-rich DCB implementations. When deploying a DCB switch in between the servers and the gateway, an Ethernet LAG is formed between the two devices, providing optimum packet distribution. In the case of the QFX3500 Switch, the Fibre Channel OxID is included in the LAG, ensuring exchange-based load balancing across the link. Additionally, for enhanced scaling, the ports of the QFX3500 can be configured in a trusted mode where it is known that there is an upstream DCB switch with FIP snooping. Increasingly, however, this option is seen as undesirable, as it adds an additional network tier and makes it hard to standardize the network access layer in a multivendor server environment. FC FC SAN FCoE Figure 7: Blade servers with embedded DCB switch and top-of-rack FCoE-FC gateway Blade Servers with Embedded FCoE-FC Gateway Typically, embedded switches have a limited power and heat budget, so the simpler the module the better. There is also an issue with limited space for port connections. With a gateway, some of these ports must be Ethernet and some must be Fibre Channel, further restricting the available bandwidth in both cases. In addition, such modules are not commonly available for all blade server families, making the deployment of a standard and consistent infrastructure challenging. Overall, these issues make this is an undesirable use case. Servers Connected Through FCoE Transit Switch to an FCoE-Enabled Fibre Channel SAN Fabric There is an interesting case for using FCoE transit switches as the access layer connecting both to Ethernet aggregation and to an FCoE-enabled Fibre Channel SAN fabric. An FCoE-FC gateway has to be actively managed and monitored by both the SAN and LAN teams—a considerable challenge for some organizations. An FCoE transit switch is not active at the FCoE layer of the protocol stack, so there is nothing for the SAN team to actively manage. Therefore, while the SAN team would still need monitoring capabilities, there is no active overlap of management, and this minimizes the possibility of configuration mistakes by different groups. 10 Copyright © 2011, Juniper Networks, Inc.
- 11. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch There are various ways to enable the FC SAN fabric. One model is to include some FCoE-enabled switches within the SAN fabric; this can be accomplished by adding an FCoE blade to one of the chassis-based SAN directors. As with the previous use cases, the FCoE ports deployed on the SAN fabric must support multiple virtual fabric ports per physical port for this deployment to be viable. Another option is to use the QFX3500 Switch configured as an FCoE-FC gateway, which is connected locally to a pure FC SAN fabric and administered by the SAN team. For larger customers, where the merging of LAN and SAN network teams is unlikely to happen for several years, this provides a very clean and simply converged deployment model. Managed by SAN Team Managed by SAN Team FC FC SAN SAN Managed by LAN Team FCoE FCoE Managed by Server Team Figure 8: Servers connected to FCoE transit switch through to an FCoE-enabled FC SAN fabric The Standards that Allow for Server I/O and Access-Layer Convergence Enhancements to Ethernet for Converged Data Center Networks—DCB Ethernet, originally developed to handle traffic using a best-effort delivery approach, has mechanisms to support lossless traffic through 802.3X Pause, but these are rarely deployed. When used in a converged network, Pause frames can lead to cross-traffic blocking and congestion. Ethernet also has mechanisms to support fine-grained queuing (user priorities), but again, these are rarely deployed within the data center. The next logical step for Ethernet will be to leverage these capabilities and enhance existing standards to meet the needs of convergence and virtualization, propelling Ethernet into the forefront as the preeminent infrastructure for LANs, SANs, and high-performance computing (HPC) clusters. These enhancements benefit Ethernet I/O convergence (remembering that most servers have multiple 1GbE network interface cards not for bandwidth but to support multiple network services), and existing Ethernet- and IP-based storage protocols such as network access server (NAS) and iSCSI. These enhancements also provide the appropriate platform for supporting FCoE. In the early days when these standards were being developed and before they moved under the auspices of the IEEE, the term Converged Enhanced Ethernet (CEE) was used to identify them. Copyright © 2011, Juniper Networks, Inc. 11
- 12. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch DCB—a set of IEEE standards. Ethernet needed a variety of enhancements to support I/O, network convergence, and server virtualization. Server virtualization is covered in other Juniper white papers, even though it is part of the DCB protocol set. With respect to I/O and network convergence, the development of new standards began with the following existing standards: 1. User Priority for Class of Service—802.1p—which already allows identification of eight separate lanes of traffic (used as-is) 2. Ethernet Flow Control (Pause, symmetric, and/or asymmetric flow control)—802.3X—which is leveraged for priority flow control (PFC) 3. MAC Control Frame for PFC—802.3bd—to allow 802.3X to apply to individual user priorities (modified) A number of new standards that leverage these components have been developed and have either been formally approved or are in the final stages of the approval process. These include: 1. PFC—IEEE 802.1Qbb—which applies traditional 802.3X Pause to individual priorities instead of the port 2. Enhanced Transmission Selection (ETS)—IEEE 802.1Qaz—which is a grouping of priorities and bandwidth allocation to those groups 3. Ethernet Congestion Management (QCN)—IEEE 802.1Qau—which is a cross network as opposed to a point-to-point backpressure mechanism 4. Data Center Bridging Exchange Protocol (DCBx), part of the ETS standard for DCB auto-negotiation The final versions of the standards specify minimum requirements for compliance, detail the maximum in terms of external requirements, and also describe in some detail the options for implementing internal behavior and the downside of some lower cost but standards-compliant ways of implementing DCB. It is important to note that these standards are separate from the efforts to solve the Layer 2 multipathing issues that are not technically necessary to make convergence work. Also, neither these standards nor those around L2 multipathing address a number of other challenges that arise when networks are converged and flattened. PFC TX Queue 0 RX Buffer 0 PFC ON RX Buffer 0 TX Queue 0 ON PFC TX Queue 1 RX Buffer 1 PFC ON RX Buffer 1 TX Queue 1 ON S Physical Port – PFC Physical Port – PFC PFC TX Queue 2 RX Buffer 2 PFC ON RX Buffer 2 T pause TX Queue 2 ON PFC TX Queue 3 O RX Buffer 3 PFC ON RX Buffer 3 P TX Queue 3 ON PFC TX Queue 4 RX Buffer 4 PFC OFF RX Buffer 4 TX Queue 4 OFF PFC TX Queue 5 RX Buffer 5 PFC OFF RX Buffer 5 TX Queue 5 OFF PFC TX Queue 6 Keeps sending DROP RX Buffer 6 PFC OFF RX Buffer 6 TX Queue 6 OFF PFC TX Queue 7 RX Buffer 7 PFC ON RX Buffer 7 TX Queue 7 ON 1 2 3 Physical Port – ETS Class Group 1 TX Queue 0 1 2 2 TX Queue 1 Class Group 2 TX Queue 2 TX Queue 3 2 5 5 2 6 5 TX Queue 4 TX Queue 5 Class Group 3 2 3 3 TX Queue 6 TX Queue 7 2 4 3 T1 T2 T3 T1 T2 T3 Offered Traffic Realized Traffic Figure 9: PFC ETS and QCN 12 Copyright © 2011, Juniper Networks, Inc.
- 13. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch Enhancements to Fibre Channel for Converged Data Center Networks—FCoE FCoE—the protocol developed within T11. The proposed FCoE protocol has been developed by the T11 Technical Committee—a subgroup of the International Committee for Information Technology Standards (INCITS)—as part of the Fibre Channel Backbone 5 (FC-BB-5) project. The standard was passed over to INCITS for public comment and final ratification in 2009, and has since been formerly ratified. In 2009, T11 started development work on Fibre Backbone 6 (FC-BB-6), which is intended to address a number of issues not covered in the first standard, and develop a number of new deployment scenarios. FCoE was designed to allow organizations to move to Ethernet-based storage while, at least in theory, minimizing the cost of change. To the storage world, FCoE is, in many ways, just FC with a new physical media type; many of the tools and services remain the same. To the Ethernet world, FCoE is just another upper level protocol riding over Ethernet. The FC-BB-5 standard clearly defines all of the details involved in mapping FC through an Ethernet layer whether directly or through simplified L2 connectivity. It lays out both the responsibilities of the FCoE-enabled endpoints and FC fabrics as well as of the Ethernet layer. Finally, it clearly states the additional security mechanisms that are recommended to maintain the level of security that a physically separate SAN traditionally provides. Overall, apart from the scale-up and scale-down aspects, FC-BB-5 defines everything needed to build and support the products and solutions discussed earlier. While the development of FCoE as an industry standard will bring the deployment of unified data center infrastructures closer to reality, FCoE by itself is not enough to complete the necessary convergence. Many additional enhancements to Ethernet and changes to the way networking products are designed and deployed are required to make it a viable, useful, and pragmatic implementation. Many, though not all, of the additional enhancements are provided by the standards developed through the IEEE DCB committee. In theory, the combination of the DCB and FCoE standards allows for full network convergence. In reality, they only solve the problem for relatively small-scale data centers. The challenge of applying these techniques to larger deployments involves the use of these protocols purely for server- and access-layer I/O convergence through the use of FCoE transit switches (DCB switches with FIP snooping) and FCoE-FC gateways (using N_Port ID Virtualization to eliminate SAN scaling and heterogeneous support issues). Juniper Networks EX4500 Ethernet Switch and QFX3500 Switch both support an FCoE transit switch mode. The QFX3500 also supports FCoE-FC gateway mode. These products are industry firsts in many ways: 1. The EX4500 and QFX3500 are fully standards-based with rich implementations from both a DCB and FC-BB-5 perspective. 2. The EX4500 and QFX3500 are purpose-built FCoE transit switches. 3. QFX3500 is a purpose-built FCoE-FC gateway which includes fungible combined Ethernet/Fibre Channel ports. 4. QFX3500 features a single Packet Forwarding Engine (PFE) design. 5. The EX4500 and QFX3500 switches both include feature-rich L3 capabilities. 6. QFX3500 supports low latency with cut-through switching. Future Direction for FCoE There are two key initiatives underway within FC-BB-6, which will prove critical to the adoption of FCoE for small and large businesses alike. For smaller businesses, a new FCoE mode has been developed, allowing for a fully functional FCoE deployment without the need for either the traditional FC services stack or FC L3 forwarding. Instead, the FCoE end devices directly discover and attach to each other through a pure L2 Ethernet infrastructure. This can be as simple as a DCB-enabled Ethernet switch, with the addition of FIP snooping for security. It makes FCoE simpler than either iSCSI or NAS, since it no longer needs a complex Fibre Channel (or FCoE) switch, and because the FCoE endpoints have proper discovery mechanisms. This mode of operation is commonly referred to as VN_Node to VN_Node or VN2VN. It can be used by itself for small to medium scale FCoE deployments, or in conjunction with the existing FCoE models for larger deployments to allow them to benefit from local L2 connectivity. Copyright © 2011, Juniper Networks, Inc. 13
- 14. WHITE PAPER - FCoE Convergence at the Access Layer with Juniper Networks QFX3500 Switch For larger businesses, a set of approaches is being investigated to remove the practical FC scaling restrictions that currently limit deployment sizes. As this work continues, it is hoped that the standards will evolve not only to solve some of these scaling limitations, but also to more fully address many of the other challenges that arise as a result of blending L2 switching, L3 FC forwarding, and FC services. Juniper fully understands these challenges, which are similar to the challenges of blending L2 Ethernet, L3 IP forwarding, and higher level network services for routing. As part of Juniper’s 3-2-1 data center architecture, we have already demonstrated many of these approaches with Juniper Networks EX Series Ethernet Switches, MX Series 3D Universal Edge Routers, SRX Series Services Gateways, and Juniper Networks Junos® Space. A Brief Note on iSCSI Although not the subject of this white paper, it is important to note that the implementation of DCB and products—such as the QFX3500 and Juniper Networks QFabric™ family of products, along with the latest generation of CNAs and storage— provides many benefits to iSCSI for those deployments where the FC-BB-5 standards prove too limiting. This is of particular interest given that most CNAs and many storage subsystems can now be deployed through different licensing as either FCoE or iSCSI, giving the end user significant protection against the protocol debate. Conclusion Juniper Networks® QFX3500 Switch is the first top-of-rack switch built to solve all of the challenges posed by access-layer convergence. It is the first fully FC-BB-5-enabled gateway capable of easily supporting upstream DCB switches, including third-party embedded blade shelf switches. It works for both rack-mount and blade servers, and for organizations with combined or separate LAN and storage area network (SAN) teams. It is also the first product to leverage a new generation of powerful ASICs. Industry firsts in many ways, Juniper Networks EX4500 Ethernet Switch and QFX3500 Switch both support an FCoE transit switch mode, and the QFX3500 also supports FCoE-FC gateway mode. They are fully standards-based with rich implementations from both a DCB and FC-BB-5 perspective and feature-rich L3 capabilities. The QFX3500 Switch is a purpose-built FCoE-FC gateway which includes fungible combined Ethernet/FC ports, a single PFE design, and low latency cut-through switching. There are a number of very practical server I/O access-layer convergence topologies that can be used as a step along the path to full network convergence. During 2011 and 2012, further events such as LAN on motherboard (LoM), QSFP, 40GbE, and the FCoE Direct Discovery Direct Attach model will further bring Ethernet economics to FCoE convergence efforts. About Juniper Networks Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers, Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking. The company serves customers and partners worldwide. Additional information can be found at www.juniper.net. Corporate and Sales Headquarters APAC Headquarters EMEA Headquarters To purchase Juniper Networks solutions, Juniper Networks, Inc. Juniper Networks (Hong Kong) Juniper Networks Ireland please contact your Juniper Networks 1194 North Mathilda Avenue 26/F, Cityplaza One Airside Business Park representative at 1-866-298-6428 or Sunnyvale, CA 94089 USA 1111 King’s Road Swords, County Dublin, Ireland authorized reseller. Phone: 888.JUNIPER (888.586.4737) Taikoo Shing, Hong Kong Phone: 35.31.8903.600 or 408.745.2000 Phone: 852.2332.3636 EMEA Sales: 00800.4586.4737 Fax: 408.745.2100 Fax: 852.2574.7803 Fax: 35.31.8903.601 www.juniper.net Copyright 2011 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice. 2000422-001-EN July 2011 Printed on recycled paper 14 Copyright © 2011, Juniper Networks, Inc.