Evaluation Copy
Compute Express Link
TM
(CXL
TM
)
Specification
October 2020
Revision 2.0
Evaluation Copy
- 1 -
C6124.0003 BN/FCURCI 39895222v1
LEGAL NOTICE FOR THIS PUBLICLY-AVAILABLE SPECIFICATION FROM COMPUTE EXPRESS LINK CONSORTIUM, INC.
© 2019-2020 COMPUTE EXPRESS LINK CONSORTIUM, INC. ALL RIGHTS RESERVED.
This CXL Specification Revision 1.1 (this “CXL Specification” or this “document”) is owned by and is proprietary to Compute Express Link
Consortium, Inc., a Delaware nonprofit corporation (sometimes referred to as “CXL” or the “CXL Consortium” or the “Company”) and/or its
successors and assigns.
NOTICE TO USERS WHO ARE MEMBERS OF THE CXL CONSORTIUM:
If you are a Member of the CXL Consortium (sometimes referred to as a “CXL Member”), and even if you have received this publicly-available version
of this CXL Specification after agreeing to CXL Consortium’s Evaluation Copy Agreement (a copy of which is available
https://www.computeexpresslink.org/download-the-specification, each such CXL Member must also be in compliance with all of the following CXL
Consortium documents, policies and/or procedures (collectively, the “CXL Governing Documents”) in order for such CXL Member’s use and/or
implementation of this CXL Specification to receive and enjoy all of the rights, benefits, privileges and protections of CXL Consortium membership: (i)
CXL Consortium’s Intellectual Property Policy; (ii) CXL Consortium’s Bylaws; (iii) any and all other CXL Consortium policies and procedures; and (iv)
the CXL Member’s Participation Agreement.
NOTICE TO NON-MEMBERS OF THE CXL CONSORTIUM:
If you are not a CXL Member and have received this publicly-available version of this CXL Specification, your use of this document is subject to your
compliance with, and is limited by, all of the terms and conditions of the CXL Consortium’s Evaluation Copy Agreement (a copy of which is available at
https://www.computeexpresslink.org/download-the-specification).
In addition to the restrictions set forth in the CXL Consortium’s Evaluation Copy Agreement, any references or citations to this document must
acknowledge the Compute Express Link Consortium, Inc.’s sole and exclusive copyright ownership of this CXL Specification. The proper copyright
citation or reference is as follows: “© 2019-2020 COMPUTE EXPRESS LINK CONSORTIUM, INC. ALL RIGHTS RESERVED.” When making
any such citation or reference to this document you are not permitted to revise, alter, modify, make any derivatives of, or otherwise amend the referenced
portion of this document in any way without the prior express written permission of the Compute Express Link Consortium, Inc.
Except for the limited rights explicitly given to a non-CXL Member pursuant to the explicit provisions of the CXL Consortium’s Evaluation Copy
Agreement which governs the publicly-available version of this CXL Specification, nothing contained in this CXL Specification shall be deemed as
granting (either expressly or impliedly) to any party that is not a CXL Member: (ii) any kind of license to implement or use this CXL Specification or any
portion or content described or contained therein, or any kind of license in or to any other intellectual property owned or controlled by the CXL
Consortium, including without limitation any trademarks of the CXL Consortium.; or (ii) any benefits and/or rights as a CXL Member under any CXL
Governing Documents.
LEGAL DISCLAIMERS FOR ALL PARTIES:
THIS DOCUMENT AND ALL SPECIFICATIONS AND/OR OTHER CONTENT PROVIDED HEREIN IS PROVIDED ON AN “AS IS” BASIS. TO
THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, COMPUTE EXPRESS LINK CONSORTIUM, INC. (ALONG WITH THE
CONTRIBUTORS TO THIS DOCUMENT) HEREBY DISCLAIM ALL REPRESENTATIONS, WARRANTIES AND/OR COVENANTS, EITHER
EXPRESS OR IMPLIED, STATUTORY OR AT COMMON LAW, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, VALIDITY, AND/OR NON-INFRINGEMENT.
In the event this CXL Specification makes any references (including without limitation any incorporation by reference) to another standard’s setting
organization’s or any other party’s (“Third Party”) content or work, including without limitation any specifications or standards of any such Third Party
(“Third Party Specification”), you are hereby notified that your use or implementation of any Third Party Specification: (i) is not governed by any of
the CXL Governing Documents; (ii) may require your use of a Third Party’s patents, copyrights or other intellectual property rights, which in turn may
require you to independently obtain a license or other consent from that Third Party in order to have full rights to implement or use that Third Party
Specification; and/or (iii) may be governed by the intellectual property policy or other policies or procedures of the Third Party which owns the Third
Party Specification. Any trademarks or service marks of any Third Party which may be referenced in this CXL Specification is owned by the respective
owner of such marks.
NOTICE TO ALL PARTIES REGARDING THE PCI-SIG UNIQUE VALUE PROVIDED IN THIS CXL SPECIFICATION:
NOTICE TO USERS: THE UNIQUE VALUE THAT IS PROVIDED IN THIS CXL SPECIFICATION IS FOR USE IN VENDOR DEFINED
MESSAGE FIELDS, DESIGNATED VENDOR SPECIFIC EXTENDED CAPABILITIES, AND ALTERNATE PROTOCOL NEGOTIATION ONLY
AND MAY NOT BE USED IN ANY OTHER MANNER, AND A USER OF THE UNIQUE VALUE MAY NOT USE THE UNIQUE VALUE IN A
MANNER THAT (A) ALTERS, MODIFIES, HARMS OR DAMAGES THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-
SIG ECOSYSTEM OR ANY PORTION THEREOF, OR (B) COULD OR WOULD REASONABLY BE DETERMINED TO ALTER, MODIFY,
HARM OR DAMAGE THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-SIG ECOSYSTEM OR ANY PORTION
THEREOF (FOR PURPOSES OF THIS NOTICE, “PCI-SIG ECOSYSTEM” MEANS THE PCI-SIG SPECIFICATIONS, MEMBERS OF PCI-SIG
AND THEIR ASSOCIATED PRODUCTS AND SERVICES THAT INCORPORATE ALL OR A PORTION OF A PCI-SIG SPECIFICATION AND
EXTENDS TO THOSE PRODUCTS AND SERVICES INTERFACING WITH PCI-SIG MEMBER PRODUCTS AND SERVICES).
Evaluation Copy
Contents
Compute Express Link Specification
October 26, 2020 3
Revision 2.0, Version 1.0
Contents
1.0 Introduction ............................................................................................................................................................................................27
1.1 Audience ....................................................................................................................................................................................27
1.2 Terminology / Acronyms ....................................................................................................................................................27
1.3 Reference Documents..........................................................................................................................................................30
1.4 Motivation and Overview....................................................................................................................................................30
1.4.1 Compute Express Link.......................................................................................................................................30
1.4.2 Flex Bus....................................................................................................................................................................32
1.5 Flex Bus Link Features .........................................................................................................................................................35
1.6 Flex Bus Layering Overview...............................................................................................................................................35
1.7 Document Scope....................................................................................................................................................................37
2.0 Compute Express Link System Architecture...........................................................................................................................39
2.1 Type 1 CXL Device.................................................................................................................................................................39
2.2 Type 2 CXL Device.................................................................................................................................................................40
2.2.1 Bias Based Coherency Model .........................................................................................................................41
2.2.1.1 Host Bias ...........................................................................................................................................42
2.2.1.2 Device Bias .......................................................................................................................................42
2.2.1.3 Mode Management.......................................................................................................................43
2.2.1.4 Software Assisted Bias Mode Management.......................................................................43
2.2.1.5 Hardware Autonomous Bias Mode Management............................................................43
2.3 Type 3 CXL Device.................................................................................................................................................................44
2.4 Multi Logical Device ..............................................................................................................................................................44
2.4.1 LD-ID for CXL.io and CXL.mem ......................................................................................................................45
2.4.1.1 LD-ID for CXL.mem.......................................................................................................................45
2.4.1.2 LD-ID for CXL.io .............................................................................................................................45
2.4.2 Pooled Memory Device Configuration Registers ...................................................................................45
2.5 CXL Device Scaling ................................................................................................................................................................47
3.0 Compute Express Link Transaction Layer................................................................................................................................48
3.1 CXL.io...........................................................................................................................................................................................48
3.1.1 CXL.io Endpoint....................................................................................................................................................49
3.1.2 CXL Power Management VDM Format.......................................................................................................50
3.1.2.1 Credit and PM Initialization.......................................................................................................54
3.1.3 CXL Error VDM Format ......................................................................................................................................55
3.1.4 Optional PCIe Features Required for CXL.................................................................................................56
3.1.5 Error Propagation................................................................................................................................................56
3.1.6 Memory Type Indication on ATS...................................................................................................................56
3.1.7 Deferrable Writes.................................................................................................................................................57
3.2 CXL.cache ..................................................................................................................................................................................58
3.2.1 Overview..................................................................................................................................................................58
3.2.2 CXL.cache Channel Description.....................................................................................................................59
3.2.2.1 Channel Ordering..........................................................................................................................59
3.2.2.2 Channel Crediting .........................................................................................................................59
3.2.3 CXL.cache Wire Description............................................................................................................................60
3.2.3.1 D2H Request ...................................................................................................................................60
3.2.3.2 D2H Response ................................................................................................................................61
3.2.3.3 D2H Data...........................................................................................................................................61
3.2.3.4 H2D Request ...................................................................................................................................62
3.2.3.5 H2D Response ................................................................................................................................62
3.2.3.6 H2D Data...........................................................................................................................................63
3.2.4 CXL.cache Transaction Description .............................................................................................................63
3.2.4.1 Device to Host Requests ............................................................................................................63
3.2.4.2 Device to Host Response...........................................................................................................74
Evaluation Copy
Contents
Compute Express Link Specification
October 26, 2020 4
Revision 2.0, Version 1.0
3.2.4.3 Host to Device Requests ............................................................................................................76
3.2.4.4 Host to Device Response...........................................................................................................77
3.2.5 Cacheability Details and Request Restrictions........................................................................................79
3.2.5.1 GO-M Responses...........................................................................................................................79
3.2.5.2 Device/Host Snoop-GO-Data Assumptions ......................................................................79
3.2.5.3 Device/Host Snoop/WritePull Assumptions .....................................................................79
3.2.5.4 Snoop Responses and Data Transfer on CXL.cache Evicts.........................................80
3.2.5.5 Multiple Snoops to the Same Address ................................................................................80
3.2.5.6 Multiple Reads to the Same Cache Line..............................................................................80
3.2.5.7 Multiple Evicts to the Same Cache Line ..............................................................................80
3.2.5.8 Multiple Write Requests to the Same Cache Line...........................................................80
3.2.5.9 Normal Global Observation (GO)............................................................................................80
3.2.5.10 Relaxed Global Observation (FastGO)..................................................................................81
3.2.5.11 Evict to Device-Attached Memory .........................................................................................81
3.2.5.12 Memory Type on CXL.cache.....................................................................................................81
3.2.5.13 General Assumptions ..................................................................................................................81
3.2.5.14 Buried Cache State Rules...........................................................................................................82
3.3 CXL.mem....................................................................................................................................................................................83
3.3.1 Introduction ...........................................................................................................................................................83
3.3.2 QoS Telemetry for Memory.............................................................................................................................84
3.3.2.1 QoS Telemetry Overview...........................................................................................................84
3.3.2.2 Reference Model for Host Support of QoS Telemetry..................................................85
3.3.2.3 Memory Device Support for QoS Telemetry.....................................................................86
3.3.3 M2S Request (Req) ..............................................................................................................................................94
3.3.4 M2S Request with Data (RwD) ........................................................................................................................97
3.3.5 S2M No Data Response (NDR)........................................................................................................................98
3.3.6 S2M Data Response (DRS) ...............................................................................................................................99
3.3.7 Forward Progress and Ordering Rules .................................................................................................... 100
3.4 Transaction Ordering Summary ................................................................................................................................... 101
3.5 Transaction Flows to Device-Attached Memory.................................................................................................... 103
3.5.1 Flows for Type 1 and Type 2 Devices ...................................................................................................... 103
3.5.1.1 Notes and Assumptions .......................................................................................................... 103
3.5.1.2 Requests from Host................................................................................................................... 104
3.5.1.3 Requests from Device in Host and Device Bias............................................................. 109
3.5.2 Type 2 and Type 3 Memory Flows ............................................................................................................ 112
3.5.2.1 Speculative Memory Read...................................................................................................... 112
3.6 Flows for Type 3 Devices................................................................................................................................................. 113
4.0 Compute Express Link Link Layers........................................................................................................................................... 115
4.1 CXL.io Link Layer ................................................................................................................................................................. 115
4.2 CXL.mem and CXL.cache Common Link Layer....................................................................................................... 117
4.2.1 Introduction ........................................................................................................................................................117
4.2.2 High-Level CXL.cache/CXL.mem Flit Overview ................................................................................... 119
4.2.3 Slot Format Definition .................................................................................................................................... 124
4.2.3.1 H2D and M2S Formats............................................................................................................. 125
4.2.3.2 D2H and S2M Formats............................................................................................................. 131
4.2.4 Link Layer Registers......................................................................................................................................... 139
4.2.5 Flit Packing Rules.............................................................................................................................................. 139
4.2.6 Link Layer Control Flit..................................................................................................................................... 141
4.2.7 Link Layer Initialization................................................................................................................................... 145
4.2.8 CXL.cache/CXL.mem Link Layer Retry..................................................................................................... 146
4.2.8.1 LLR Variables................................................................................................................................ 147
4.2.8.2 LLCRD Forcing ............................................................................................................................. 149
4.2.8.3 LLR Control Flits ......................................................................................................................... 151
4.2.8.4 RETRY Framing Sequences.................................................................................................... 151
4.2.8.5 LLR State Machines ................................................................................................................... 152
4.2.8.6 Interaction with Physical Layer Reinitialization............................................................. 156
Evaluation Copy
Contents
Compute Express Link Specification
October 26, 2020 5
Revision 2.0, Version 1.0
4.2.8.7 CXL.cache/CXL.mem Flit CRC ............................................................................................... 157
4.2.9 Poison and Viral ................................................................................................................................................158
4.2.9.1 Viral .................................................................................................................................................. 158
5.0 Compute Express Link ARB/MUX.............................................................................................................................................. 160
5.1 Virtual LSM States............................................................................................................................................................... 161
5.1.1 Additional Rules for Local vLSM Transitions ........................................................................................ 164
5.1.2 Rules for Virtual LSM State Transitions Across Link.......................................................................... 164
5.1.2.1 General Rules............................................................................................................................... 164
5.1.2.2 Entry to Active Exchange Protocol ..................................................................................... 164
5.1.2.3 Status Synchronization Protocol.........................................................................................165
5.1.2.4 State Request ALMP.................................................................................................................. 167
5.1.2.5 State Status ALMP ..................................................................................................................... 169
5.1.2.6 Unexpected ALMPs ................................................................................................................... 171
5.1.3 Applications of the vLSM State Transition Rules................................................................................ 172
5.1.3.1 Initial Link Training .................................................................................................................... 172
5.1.3.2 Status Exchange Snapshot Example.................................................................................. 175
5.1.3.3 L1 Abort Example....................................................................................................................... 176
5.2 ARB/MUX Link Management Packets......................................................................................................................... 177
5.2.1 ARB/MUX Bypass Feature............................................................................................................................. 178
5.3 Arbitration and Data Multiplexing/Demultiplexing .............................................................................................. 179
6.0 Flex Bus Physical Layer .................................................................................................................................................................. 180
6.1 Overview ................................................................................................................................................................................. 180
6.2 Flex Bus.CXL Framing and Packet Layout.................................................................................................................181
6.2.1 Ordered Set Blocks and Data Blocks........................................................................................................181
6.2.2 Protocol ID[15:0]............................................................................................................................................... 182
6.2.3 x16 Packet Layout ............................................................................................................................................ 183
6.2.4 x8 Packet Layout............................................................................................................................................... 184
6.2.5 x4 Packet Layout............................................................................................................................................... 187
6.2.6 x2 Packet Layout............................................................................................................................................... 187
6.2.7 x1 Packet Layout............................................................................................................................................... 187
6.2.8 Special Case: CXL.io -- When a TLP Ends on a Flit Boundary........................................................ 187
6.2.9 Framing Errors ................................................................................................................................................... 188
6.3 Link Training.......................................................................................................................................................................... 190
6.3.1 PCIe vs Flex Bus.CXL Mode Selection ...................................................................................................... 190
6.3.1.1 Hardware Autonomous Mode Negotiation..................................................................... 190
6.3.1.2 CXL 2.0 Versus CXL 1.1 Negotiation.................................................................................. 194
6.3.1.3 Flex Bus.CXL Negotiation with Maximum Supported Link
Speed of 8GT/s or 16GT/s ..................................................................................................... 196
6.3.1.4 Link Width Degradation and Speed Downgrade.......................................................... 197
6.4 Recovery.Idle and Config.Idle Transitions to L0 .................................................................................................... 197
6.5 L1 Abort Scenario ............................................................................................................................................................... 197
6.6 Exit from Recovery.............................................................................................................................................................. 197
6.7 Retimers and Low Latency Mode ................................................................................................................................. 197
6.7.1 SKP Ordered Set Frequency and L1/Recovery Entry........................................................................ 198
7.0 Switching...............................................................................................................................................................................................201
7.1 Overview ................................................................................................................................................................................. 201
7.1.1 Single VCS Switch............................................................................................................................................. 201
7.1.2 Multiple VCS Switch ........................................................................................................................................202
7.1.3 Multiple VCS Switch with MLD Ports........................................................................................................203
7.2 Switch Configuration and Composition..................................................................................................................... 204
7.2.1 CXL Switch Initialization Options............................................................................................................... 204
7.2.1.1 Static Initialization ..................................................................................................................... 204
7.2.1.2 Fabric Manager Boots First .................................................................................................... 205
7.2.1.3 Fabric Manager and Host Boot Simultaneously............................................................ 207
评论0