CXL Fabric Management Standards
- 1. What are CXL, PCI SIG, DMTF, OFA, OCP & SNIA doing to make CXL fabric, memory & device management interoperable? CXL Fabric Management Standards
- 2. CXL Fabric Management Standards Jeff Hilland, Distinguished Technologist HPE, President DMTF
- 3. • Provide a roadmap of Open Communities solving CXL fabric management • Show how these communities are working together to ensure an interoperable, multi-vendor solution is possible • Any help the community could provide to these communities would be appreciated Agenda
- 4. • How do we get to commercially available off the shelf disaggregated composability using industry standard components • Without writing millions of lines of custom code to make it work • While still allowing for differentiation The Approach: • A multitude of open The Infrastructure Management Dilemma: BMC Host w/ FW, OS/VMM Mgmt SW CXL Switch CXL Controller Media/Device CXL 2/3 FM API Agent I2C UART PCIe (B)MC (DC-SCM) Ethernet FM API Agent Enclosure Boundary (sometimes) I2C UART PCIe Ethernet ? ? ? ? ? ? ?
- 5. • CXL management is largely via the CCI & FM API • CCI (Component Command Interface) is used to manage CXL Memory • Capacity/Label Info, Health & Alerts, Media/Poison, Security, QoS, etc. • FM API are commands used to manage Switches • Also device status registers (Event/Log) • Accessed through the mailbox registers or MCTP • Two mailboxes (primary, secondary): one for system SW, one for FW (only primary is mandatory) • MCTP is for sideband/out-of-band management • This traffic is not currently comprehended by BMCs and is entirely different than the rest of platform management CXL
- 6. • PCI cards typically have both PCI VDM for management traffic • This is what MCTP uses for In-Band management • Flit mode requires changes (Non-flit mode is specified in PCI Spec) • PCI signaling still has the I2C/I3C, USB pins for side band signaling • This is used for out-of-band management using MCTP • PCI devices already support not only MCTP but many support most of the other PMCI payloads (next slide) • Since cxl.io is basically cxl encapsulated pci, cxl.io devices can be managed with PCI VDM traffic over CXL PCI SIG
- 7. • PMCI standards • PMCI is a suite of standards used for platform management • MCTP is the low level transport • SPDM flows over MCTP • PLDM flows over MCTP too • PLDM has Type 2, FW Update, RDE (Redfish Device Enablement), File Transfer and other payloads • FRU is going through a modernization • Work In Progress is out, moving from a binary with extensions to JSON payload • Every modern BMCs supports most of these protocols • There is a split in the industry between those that prefer Redfish Device Enablement vs PLDM Type 2 • https://www.dmtf.org/standards/pmci DMTF - PMCI KCS PCIe VDM Physical Layer Management Component Transport Protocol (MCTP) Transport Layer MCTPover KCS MCTPHost Interface SMBus/I2C Serial RMII Based Transport (RBT) MCTPover Serial PLDM over MCTP MCTPover PCIe VDM SPDM over MCTP NC-SI over MCTP NVMe-MI over MCTP MCTP Control Type = 0 PLDM Type = 1 NC-SI Control Type = 2 Ethernet (NC-SI Passthru) Type = 3 NVMe-MI Type = 4 SPDM Type = 5 MCTPover SMBus/I2C NC-SI Passthru 0: Messaging Control & Discovery 1: SMBIOS DataTransfer 2: Monitoring & Control 4: FRU Data 5: Firmware Update 6: Redfish DeviceEnablement 3: BIOS Control & Configuration Upper Layer PLDM Platform Level Data Model (PLDM) Message Types MCTP Message Types = Physical Layer = Binding = Transport = Data Model/Message = PLDM Message = Host Interface Secured Messages Type = 6 Sec d Msgs over MCTP Vendor Defined Messages Type = 7E/7F 0: MCTP Control 1: PLDM 2: NC-SI Control 4: NVMe-MI 5: SPDM 7E/7F: Vendor Defined 3: NC-SI Passthru MCTP Message Types Encapsulation = Encapsulation NC-SI Control 3F: OEM Specific
- 8. • Redfish is the industry platform management standard • REST protocol with a JSON payload, specified by schemas • Suite of self validation based on profiles. • Started as just sever management, but now convers storage, composability, fabrics, DCIM (power/cooling), DPUs, aggregators, operating systems, workloads, containers and CXL • CXL support released starting in the 2020.3 release • White paper, presentation, YouTube • Updates keep coming. 3.0 support complete in 2023.2 • Also has all of the device support, eventing, security, etc. • CXL Mapping Spec (“Rosetta Stone”) now available • Maps the CCI & FM API in 3.0 to Redfish • https://redfish.dmtf.org DMTF - Redfish PCIe Devices D1 PCIe Functions F1 CXL Logical Devices LD1
- 9. • OFA Sunfish group is working on an Open Fabric Management Framework • Code to manage multiple fabrics • Represent components on the fabric. • Represent management of the fabric itself as well. • Interface is based on Redfish/Swordfish OFA (Open Fabrics Alliance) - Sunfish
- 10. • OCP Hardware Management • Working in multiple areas on platform management proscriptive standards • Profiles – Redfish System & Component Profiles to validate conformance for Systems, Devices, Memory and Fabrics (et al) • Combined with the Redfish validation tools, the profiles can be used to programmatically validate implementations • SMC – Satellite Management Controller Specification • Requirements for fabric attached devices (MCTP, PLDM, FW Update and RDE requirements) • OCP CMS – Composable Memory Systems • Meeting to solve data plane & solutions issues • Deep Dive on CXL Redfish model in the archives OCP Efforts HW MGMT
- 11. • SNIA – Swordfish • Extension of Redfish, leveraging the core of Redfish storage & memory • Has advanced memory pooling, grouping & service models that align memory as storage with other traditional storage media • Has advanced NVMe, block, file & object support • https://www.snia.org/swordfish SNIA
- 12. • Every community has a part • Closely working together to enable solutions The Answer: BMC Host w/ FW, OS/VMM Mgmt SW CXL Switch CXL Controller Media/Device (B)MC (DC-SCM)
- 13. • Get involved in OCP: HW Mgmt, Profile sub-project & CMS • Get involved in any of the Open Communities • Contribute code to, for example, Sunfish • Never too late to help. Call to Action
- 14. Thank you!