InfiniBand
TM
Architecture
Specification Volume 1
Release 1.2.1
Copyright © 1999, 2001, 2002, 2003, 2004, 2007 by InfiniBand
SM
Trade Association.
All rights reserved.
November 2007
Final Release
InfiniBand
TM
Architecture Release 1.2.1 November 2007
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL RELEASE
InfiniBand
SM
Trade Association Page 2 Proprietary and Confidential
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
LEGAL DISCLAIMER This IBTA specification provided “AS IS” and without
any warranty of any kind, including, without limita-
tion, any express or implied warranty of non-infringe-
ment, merchantability or fitness for a particular
purpose.
In no event shall IBTA or any member of IBTA be liable
for any direct, indirect, special, exemplary, punitive,
or consequential damages, including, without limita-
tion, lost profits, even if advised of the possibility of
such damages.
Table 1 Revision History
Revision Release Date
1.0 9/26/2000 Release 1.0
1.0.a 6/19/2001 Release 1.0 augmented with errata material. Updates only
correct errors - no additional features have been added.
1.1 11/06/2002 Release 1.0.a augmented with additional features. Revised
SA and CM Class with new version.
1.2 9/7/2004 Release 1.1 augmented with additional features (added
annexes A7 through A10). Incorporated errata.
1.2.1 11/30/2004 Release 1.2 augmented with additional features (added
annex A11-A13).
InfiniBand
TM
Architecture Release 1.2.1 November 2007
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL RELEASE
InfiniBand
SM
Trade Association Page 3 Proprietary and Confidential
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
TABLE OF CONTENTS
Chapter 1: Introduction...........................................................................................................62
1.1 Acknowledgments..................................................................................................................... 62
1.2 InfiniBand Conceptual Overview...............................................................................................64
1.2.1 The Problem ........................................................................................................................... 64
1.2.2 Features.................................................................................................................................. 65
1.2.3 Benefits................................................................................................................................... 65
1.3 Scope........................................................................................................................................ 66
1.4 Document Organization ............................................................................................................ 67
1.4.1 Series of Volumes ................................................................................................................... 67
1.4.2 Volume 1 Organization............................................................................................................ 68
1.5 Document Conventions ............................................................................................................ 68
1.5.1 Byte Ordering.......................................................................................................................... 68
1.5.2 Numeric Values....................................................................................................................... 69
1.6 Disclaimer ................................................................................................................................. 70
Chapter 2: Glossary ................................................................................................................71
Chapter 3: Architectural Overview.........................................................................................88
3.1 Architecture Scope ................................................................................................................... 89
3.1.1 Topologies & Components...................................................................................................... 90
3.2 Communication......................................................................................................................... 92
3.2.1 Queuing .................................................................................................................................. 92
3.2.2 Connections............................................................................................................................ 96
3.3 Communications Stack ............................................................................................................. 96
3.4 IBA Components....................................................................................................................... 97
3.4.1 Links & Repeaters................................................................................................................... 97
3.4.2 Channel Adapters ................................................................................................................... 98
3.4.3 Switches ................................................................................................................................. 99
3.4.4 Routers ................................................................................................................................. 100
3.4.5 Management Components.................................................................................................... 101
3.4.5.1 Subnet Managers .................................................................................................. 101
3.4.5.2 Subnet Management Agents................................................................................. 102
3.4.5.3 General Service Agents ........................................................................................ 102
3.5 IBA Features........................................................................................................................... 103
3.5.1 Queue Pairs.......................................................................................................................... 103
3.5.2 Types of Service ................................................................................................................... 103
3.5.3 Keys...................................................................................................................................... 105
InfiniBand
TM
Architecture Release 1.2.1 November 2007
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL RELEASE
InfiniBand
SM
Trade Association Page 4 Proprietary and Confidential
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
3.5.4 Virtual Memory Addresses.................................................................................................... 107
3.5.5 Protection Domains............................................................................................................... 107
3.5.6 Partitions ............................................................................................................................... 108
3.5.7 Virtual Lanes......................................................................................................................... 108
3.5.8 Quality of Service.................................................................................................................. 109
3.5.8.1 Service Level......................................................................................................... 110
3.5.8.2 SL to VL mapping.................................................................................................. 110
3.5.8.3 Partitions ...............................................................................................................110
3.5.9 Injection Rate Control ........................................................................................................... 110
3.5.10 Addressing .............................................................................................................................111
3.5.11 Multicast................................................................................................................................ 113
3.5.11.1 Multicast Example ................................................................................................. 114
3.5.11.2 Group Management .............................................................................................. 115
3.5.11.3 Multicast Prune...................................................................................................... 118
3.5.12 Verbs..................................................................................................................................... 118
3.6 Channel & Memory Semantics ............................................................................................... 118
3.6.1 Communication Interface ...................................................................................................... 119
3.6.2 IBA Transport Services ......................................................................................................... 121
3.7 IBA Layered Architecture........................................................................................................ 125
3.7.1 Physical Layer....................................................................................................................... 125
3.7.2 Link Layer ............................................................................................................................. 126
3.7.3 Network Layer....................................................................................................................... 127
3.7.4 Transport Layer..................................................................................................................... 128
3.7.5 Upper Layer Protocols .......................................................................................................... 130
3.7.5.1 Subnet Management ............................................................................................. 130
3.7.5.2 General Services................................................................................................... 131
3.8 IBA Transaction Flow.............................................................................................................. 132
3.9 IBA Management Infrastructure.............................................................................................. 133
3.9.1 Management Datagrams ...................................................................................................... 137
3.9.2 Management Methods .......................................................................................................... 137
3.9.2.1 Gets & Sets ........................................................................................................... 137
3.9.2.2 Traps and Notices ................................................................................................. 137
3.9.2.3 Sends .................................................................................................................... 138
3.9.2.4 Reports.................................................................................................................. 138
3.9.3 Management Interfaces ........................................................................................................ 138
3.9.4 Subnet Management Interface.............................................................................................. 138
3.9.4.1 Fabric Initialization................................................................................................. 139
3.9.4.2 Directed Routes..................................................................................................... 139
3.9.5 General Service Interface ..................................................................................................... 140
3.9.5.1 Redirection ............................................................................................................ 140
InfiniBand
TM
Architecture Release 1.2.1 November 2007
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL RELEASE
InfiniBand
SM
Trade Association Page 5 Proprietary and Confidential
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
3.10 I/O Operation .......................................................................................................................... 140
Chapter 4: Addressing..........................................................................................................143
4.1 Terminology And Concepts.....................................................................................................144
4.1.1 GID Usage and Properties.................................................................................................... 145
4.1.2 Channel Adapter, Switch, and Router Addressing Rules...................................................... 149
4.1.3 Local Identifiers..................................................................................................................... 149
Chapter 5: Data Packet Format ............................................................................................152
5.1 Packet Types .......................................................................................................................... 152
5.2 Data Packet Format ................................................................................................................ 153
5.2.1 Local Route Header (LRH) - 8 Bytes .................................................................................... 156
5.2.2 Global Route Header (GRH) - 40 Bytes................................................................................ 156
5.2.3 Base Transport Header (BTH) - 12 Bytes............................................................................. 157
5.2.4 Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes .................................... 158
5.2.5 Datagram Extended Transport Header (DETH) - 8 Bytes.................................................... 159
5.2.6 RDMA Extended Transport Header (RETH) - 16 Bytes........................................................ 159
5.2.7 Atomic Extended Transport Header (AtomicETH) - 28 Bytes ............................................... 160
5.2.8 ACK Extended Transport Header (AETH) - 4 Bytes ............................................................. 161
5.2.9 Atomic ACK Extended Transport Header (AtomicAckETH) - 8 Bytes .................................. 161
5.2.10 Immediate Data Extended Transport Header (ImmDt) - 4 Bytes .......................................... 162
5.2.11 INVALIDATE EXTENDED TRANSPORT HEADER (IETH) - 4 BYTES ................................ 162
5.2.12 Payload................................................................................................................................. 162
5.2.13 Invariant CRC ....................................................................................................................... 162
5.2.14 Variant CRC .......................................................................................................................... 163
5.3 Raw Packet Format ................................................................................................................ 163
5.4 Packet Examples .................................................................................................................... 163
Chapter 6: Physical Layer Interface.....................................................................................165
6.1 Overview................................................................................................................................. 165
6.2 Services provided by the Physical Layer. ............................................................................... 165
6.3 Interface between physical and Link Layers........................................................................... 166
6.3.1 Interface between physical receive and link receive............................................................. 166
6.3.1.1 Phy_link - Physical Link Status.............................................................................. 166
6.3.1.2 L_Init_Train - Link Initiate Retraining..................................................................... 166
6.3.1.3 rcv_stream - Receive Stream................................................................................ 167
6.3.2 Interface between physical Transmit and link Transmit. ....................................................... 167
6.3.2.1 Xmit_stream - Transmit Stream............................................................................. 167
6.3.2.2 Xmit_Ready - Physical Transmitter Ready............................................................ 167
