Xilinx Embedded

RDMA Enabled NIC

v4.0

LogiCORE IP Product Guide

Vivado Design Suite

PG332 December 2, 2022
Table of Contents
IP Facts

Chapter 1: Overview
Navigating Content by Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Core Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Feature Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Unsupported Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Licensing and Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 2: Product Specification

RDMA Enabled NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Work Requests/Work Queue Entries (WQEs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Work Completions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
RDMA Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
ERNIC RX Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
ERNIC TX Path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Resource Utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Port Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Parameter Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Register Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Chapter 3: Designing with the Core

General Design Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Clocking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Chapter 4: Design Flow Steps

Customizing and Generating the Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Synthesis and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

ERNIC v4.0 2
PG332 December 2, 2022 www.xilinx.com
 Chapter 5: Example Design
Example Design Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Example Design Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Simulating the Example Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Example Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 6: ERNIC Software Flow

ERNIC Global Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
QP1 Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Memory Registration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
RC QP Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
WQE SQ Posting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Receiving Incoming RDMA SEND Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Processing WQE Completions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Enabling QP HW Offload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
QP Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
QP Fatal Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Appendix A: Requesting ERNIC Support Questionnaire

Appendix B: Debugging
Debugging the IP/System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Appendix C: Additional Resources and Legal Notices

Xilinx Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Documentation Navigator and Design Hubs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Please Read: Important Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

ERNIC v4.0 3
PG332 December 2, 2022 www.xilinx.com
 IP Facts

Introduction LogiCORE™ IP Facts Table

Core Specifics
The Xilinx® ERNIC™ (Embedded RDMA enabled Versal® ACAP
NIC) IP is an implementation of RDMA over Kintex UltraScale+™,
Converged Ethernet (RoCE v2) enabled NIC Supported
Virtex® UltraScale™,
Device Family (1)
functionality. This parameterizable soft IP core can Virtex UltraScale+,
work with a wide variety of Xilinx hard and soft MAC Zynq® UltraScale+
IP implementations providing a high throughput, Supported User
AXI4-Lite, AXI4, and AXI4-Stream
low latency, and completely hardware offloaded Interfaces
reliable data transfer solution over standard Resources Performance and Resource Utilization web page
Ethernet. The ERNIC IP allows simultaneous
connections to multiple remote hosts running RoCE Provided with Core
v2 traffic. Design Files Encrypted RTL
Example Design Verilog
Test Bench Not Provided
Features Constraints File Xilinx Design Constraints (XDC)
Simulation
Not Provided
• Support for RDMA functionality Model
Supported Linux kernel drivers and user space libraries for
° RoCE v2 S/W Driver(2) RDMA

° Packet retransmission on errors are handled Tested Design Flows(2)

in the hardware by the IP.
Vivado® Design Suite
Design Entry
• 100 Gb/s line rate Vivado IP integrator

• Support for Reliable Connection (RC) RDMA For supported simulators, see the
Simulation
Xilinx Design Tools: Release Notes Guide.
transport service types
Synthesis Vivado Synthesis
• QP1 support for sending and receiving MAD
packets Support
Release Notes
• Hardware handshake mode on user interface to and Known N/A
support hardware RDMA applications in the Issues
user logic All Vivado IP
Master Vivado IP Change Logs: 72775
• Supports incoming and outgoing RDMA SEND, Change Logs
RDMA READ, RDMA WRITE, RDMA SEND WITH Xilinx Support web page
IMM, RDMA WRITE WITH IMM, and RDMA
Notes:
SEND WITH INVALIDATE message types.
1. For a complete list of supported devices, see the Vivado IP
• Designed to scale up to 2047 RDMA Queue catalog.
pairs(3) 2. For the supported versions of third-party tools, see the
Xilinx Design Tools: Release Notes Guide.
• Support for IPv4 and IPv6 packets 3. For -1 speed grade devices design might have timing
violations for more than 64 QP configuration.
• Support for Explicit Congestion Notification
(ECN)
• Supports Priority flow control with different
priorities for RoCE and non-RoCE traffic.
• Supports memory registrations and protection
domains

ERNIC v4.0 4
PG332 December 2, 2022 www.xilinx.com Product Specification
 Chapter 1

Overview

Navigating Content by Design Process

Xilinx ® documentation is organized around a set of standard design processes to help you
find relevant content for your current development task. This document covers the
following design processes:

• Hardware, IP, and Platform Development: Creating the PL IP blocks for the hardware
platform, creating PL kernels, subsystem functional simulation, and evaluating the
Vivado ® timing, resource and power closure. Also involves developing the hardware
platform for system integration. Topics in this document that apply to this design
process include:

° Port Descriptions

° Register Space

° Clocking

° Resets

° Customizing and Generating the Core

° Example Design

Core Overview
This chapter provides an overview of the ERNIC IP core and details of the applications,
licensing requirements, and standards conformance. ERNIC is a soft IP implementing RDMA
over a Converged Ethernet (RoCE v2) protocol for embedded target or initiator devices. This
implementation is based on the specifications described in InfiniBand Architecture
Specification Volume 1, Annex A16 RoCE and Annex 17 RoCE V2 [Ref 1].

ERNIC v4.0 5
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

Figure 1-1 shows the ERNIC and its connections to other IPs in the subsystem.
X-Ref Target - Figure 1-1

RX Packet Handler ERNIC IP

non_roce_cmac_s_axis_* non_roce_dma_m_axis_*

rx_pkt_hndlr_ddr_m_axi_*
Request Validation
DMA rx_pkt_hndlr_o_rq_db_*
roce_cmac_s_axis_* Engine

Response rx_pkt_hndlr_rdrsp_m_axi_*
Validation

Response Handler
resp_hndler_o_send_cq_db_*
Flow Control
Manager ACK PSN Buffer
FSM Logic
cmac_m_axis_* MPSN Buffer OSQ Buffer
resp_hndler_m_axi_*

non_roce_dma_s_axis_*

WQE Proc Top QP Manager Top i_qp_sq_pidb_*

TX ACK Buffer Retransmit Logic i_qp_rq_cidb_*

Header
Generator qp_mgr_m_axi_*
CRC Caching Logic
Buffer
Logic Manager
DMA
s_axi_lite_*
Engine Configuration Registers
wqe_proc_top_m_axi_* AXI4-Stream interface
AXI MM interface
wqe_proc_wr_ddr_m_axi_*
X25376-052621

Figure 1‐1: ERNIC IP Block Diagram

Note: The user logic or target IP that connects to ERNIC is referred to as application and the
direction of the arrows is from master to slave.

Apart from the ERNIC IP, the ERNIC subsystem includes the Xilinx Ethernet IP, AXI DMA, and
AXI Interconnect among other IPs. On the user application front, the ERNIC IP exposes side
band interfaces to allow efficient doorbell exchanges without going through the
interconnect.

Each queue is identified with a set of read and write pointers called the Producer Index
(write pointer) and Consumer Index (read pointer). The register address locations for these
pointers are termed as doorbells in this document. A doorbell exchange or doorbell ringing
indicates that the corresponding register location is updated.

ERNIC v4.0 6
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

Feature Summary
The ERNIC IP interfaces with any Ethernet MAC IP using an AXI4-Stream interface. Access to
DDR or any other memory region is necessary for reading and writing various data
structures for RDMA packet processing. This connection is achieved using multiple AXI4
interfaces. The IP works on a 512-bit internal datapath that can be completely hardware
accelerated without any software intervention for data transfer. All recoverable faults like
retransmission due to packet drops are also handled entirely in the hardware.

The ERNIC IP implements embedded RNIC functionality. As a result, only the following
subset of RoCE v2 functionality is implemented compared to a general purpose RNIC:

• Support for RDMA SEND, RDMA READ, RDMA WRITE, RDMA SEND INVALIDATE, RDMA
SEND IMMEDIATE, and RDMA WRITE IMMEDIATE for incoming and outgoing packets.
Atomic operations are not supported.
• Support for up to 2046 connections.
• Scalable design of up to 2047 RDMA Queue pairs.
Note: Default Vivado strategies allow for the timing to pass up to 127 queue pairs. To match the
timing for 2047 queue pairs, use the Vivado strategy — Performance_refinePlacement.
• Supports dynamic memory registration.
• Hardware handshake mechanism for efficient doorbell exchange with the user
application logic.
Note: When switching in the handshake mode, the software layer should not have any traffic on
that particular QP.

ERNIC Modules
The ERNIC IP consists of the following main modules that are explained in this section.

• QP Manager
• WQE Processor Engine
• RX PKT Handler
• Response Handler
• Flow Control Manager

ERNIC v4.0 7
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

QP Manager
The QP Manager module houses the configurations for all the QPs and provides an
AXI4-Lite interface to the processor. It also arbitrates across various SEND Queues and
caches the SEND Work Queue Entries (WQEs). These WQEs are then provided to the WQE
processor module for further processing. This module also handles the QP pointer updates
in the event of retransmission.

WQE Processor Engine

The WQE Processor Engine reads the cached WQEs from the QP Manager module and
handles the following tasks:

• Validates the incoming WQE for any invalid opcode.

• Creates the header for the RDMA packets based on the Payload Max Transfer Unit
(PMTU) and programs internal DMA engine.
• It also triggers the DMA to start outgoing packet transfers.

The WQE Processor Engine is also responsible for sending outgoing acknowledgment
packets for the incoming RDMA SEND/WRITE requests and read responses for incoming
RDMA READ requests.

RX PKT Handler
The RX PKT Handler module receives the incoming RDMA packets. Non-RDMA packets
should be filtered out before receiving the RX PKT Handler (roce_cmac_s_axis)
interface. The ERNIC IP handles the following types of incoming RoCE v2 packets:

• RDMA SEND, RDMA WRITE, RDMA READ and response packets for RDMA READ
(request sent from ERNIC)
• RDMA SEND with Invalidate, RDMA SEND with immediate, and RDMA WRITE with
immediate packets
• Acknowledgment packets for RDMA WRITE/RDMA SEND (request sent from ERNIC)
• Communication management (Management Datagram) packets to QP1

The RX PKT Handler module is responsible for validating the incoming packets. It also
triggers outgoing acknowledgment packets for incoming RDMA SEND and RDMA WRITE
requests and pushes the packets that pass the validation to the corresponding memory
location. The RDMA READ responses are channeled to the target application directly. The
module handles the incoming RDMA READ requests and forwards the request to the TX
path.

ERNIC v4.0 8
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

The RX PKT Handler module also decodes the RDMA SEND invalidate/Immediate and RDMA
WRITE Immediate packets. The 32-bit data present in either IETH or IMMDT headers is
provided on a separate AXI4-Stream interface. 64 bits of data is provided on this streaming
interface for every entry. Below table shows encoding of these 64 bits.

Table 1‐1: Invalidate/Immediate Data Entry Structure

Bits Field Description
Contents of this field is defined by the Type
31:0 ImmDt/R_KEY field. This field contains the immediate data or
R_KEY in case of send with invalidate.
This field indicates the type of the incoming
packet that triggered this completion.
5’h01 = SEND INVALIDATE
36:32 Type
5’h02 = SEND IMMEDIATE
5’h03 = WRITE_IMMEDIATE
5’h04 to 5’h1F = Reserved
47:37 QPID QP identifier which triggered this entry
This field is valid only when Type field indicates
SEND with immediate or SEND with invalidate
55:48 RQ PI pointer commands. This field indicates the receive
queue pointer where corresponding SEND
command is located.
63:56 Reserved Reserved

An external hardware logic needs to be implemented to handle this invalidate/immediate

data provided on this interface. For example AXI DMA IP to convert streaming interface to
AXI Memory mapped interface and write the data to specific location and notify to software
or hardware application.

Outgoing Pause requests for RDMA traffic are handled inside this module. When remaining
buffer locations reaches to XON condition then it triggers for Pause ON and deasserts when
buffer pointer reaches to XOFF condition.

Response Handler
The Response Handler module manages the outstanding queues. These queues hold the
information about all packets sent to the remote host but have not yet been acknowledged
or responded to. In addition, this module triggers a re-transmission if the remote host
sends a Negative Acknowledgment (NAK). If this module does not receive a response from
the remote host within a specified time (timeout value), it triggers a timeout related
retransmission.

ERNIC v4.0 9
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

Retransmission is triggered either due to an incoming NACK or an internal timeout. When

a retransmission is triggered, ERNIC IP takes care of the following:

1. Identifies the PSN and MSN values to be retransmitted.

2. Rolls back the PSN and MSN values for that queue pair.
3. Rolls back the current SQ pointer to the WQE that needs to be retransmitted.
4. ERNIC IP supports partial WQE retransmission, it retransmits from the PSN for the
incoming NACK.

Flow Control Manager

This module maintains separate buffers in RX path and generates PFC control signals as per
the programmable full conditions of those buffers. Similarly in TX path, it also has two
separate buffers for RoCE and non-RoCE. On the assertion of any Pause signals, the module
will stop reading from the buffer.

Applications
The ERNIC IP can be used in range of applications that require reliable transfer of packets
across the network fabric. A few such applications are listed here:

• Sensor Data Acquisition and transfer over RoCE V2

• Video/Image capture and transfer over RoCE V2
• Remote storage nodes over RoCE V2

Unsupported Features
The ERNIC IP does not support:

• Incoming ATOMIC operations

• Outgoing ATOMIC operations
• Incoming/outgoing RDMA SEND packets of 0 length
• Maximum DMA length for RDMA payload more than 8 MB
• Incoming RDMA READ Requests with DMA Length equal to or less than 4 bytes
• Incoming RDMA READ/WRITE Requests with virtual address not on 64-byte boundary
• Incoming READ Response and WRITE Request Length non-multiple of 64-byte

ERNIC v4.0 10
PG332 December 2, 2022 www.xilinx.com
 Chapter 1: Overview

Licensing and Ordering

This Xilinx LogiCORE™ IP module is provided under the terms of the Xilinx Core License
Agreement. The module is shipped as part of the Vivado® Design Suite. For full access to
all core functionalities in simulation and in hardware, you must purchase a license for the
core. To generate a full license, visit the product licensing web page. Evaluation licenses and
hardware timeout licenses might be available for this core or subsystem. Contact your local
Xilinx sales representative for information about pricing and availability.

License Checkers
If the IP requires a license key, the key must be verified. The Vivado design tools have
several license checkpoints for gating licensed IP through the flow. If the license check
succeeds, the IP can continue generation. Otherwise, generation halts with an error. License
checkpoints are enforced by the following tools:

• Vivado synthesis
• Vivado implementation
• write_bitstream (Tcl command)

IMPORTANT: IP license level is ignored at checkpoints. The test confirms a valid license exists. It does
not check IP license level.

ERNIC v4.0 11
PG332 December 2, 2022 www.xilinx.com
 Chapter 2

Product Specification
The ERNIC IP provides an embedded implementation of a RoCE v2 enabled NIC. The RDMA
technology allows for faster movement of data over standard Ethernet while completely
offloading CPU bandwidth. The ERNIC IP core comes with SW drivers that can be ported to
any Zynq® MPSoC or FPGA devices. This allows the ERNIC IP to function independent of
any external processor.

RDMA Enabled NIC

The Xilinx® Embedded RDMA enabled NIC (ERNIC) controller can interface with any user
application using one of the following methods:

• Closely integrated HW handshake mechanism which provides a low latency, light

weight interface to the target system
• OFED-compliant RDMA hardware drivers and user space libraries providing standard
SW API to post work requests to ERNIC

Figure 2-1 shows sample end-to-end system with multiple host CPUs and multiple native
NVMe devices talking over a network fabric through the Xilinx ERNIC + NVMEOFABRIC IP
subsystem.
X-Ref Target - Figure 2-1

CPU RNIC PCIe NVMe

RNIC Xilinx PCIe
1x100G SoC NVMe
PCIe
PCIe PCIe
Ethernet Switch
CPU RNIC PCIe
RNIC Xilinx PCIe
2x100G SoC PCIe
CPU RNIC
PCIe NVMe

X19877-120220

Figure 2‐1: Xilinx ERNIC + NVMe over Interconnect

The host side RDMA enabled NICs (RNIC) shown in Figure 2-1 need to support RoCE v2
technology. The following sections describe the key data structures used in the ERNIC
implementation.

ERNIC v4.0 12
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Work Requests/Work Queue Entries (WQEs)

Work Requests are used to submit units of work to the ERNIC IP. The operations which can
be posted to the Send Work Queues by the application are:

• RDMA Write
• RDMA WRITE Immediate
• RDMA SEND
• RDMA SEND Immediate
• RDMA Read
• RDMA SEND Invalidate

The following work requests are not allowed:

• ATOMIC
• Bind Memory Window
• Local Invalidate
• Fast Register Physical MR

The Receive Queue work requests need not be posted by application as the ERNIC
Hardware automatically re-posts consumed receive buffers as per the configured receive
queue depth.

Table 2-1 shows the structure of Send work requests. Each Work Queue Entry (WQE) is 64
bytes in size.

Table 2‐1: WQE Structure

Bitwidth Content Size (B) Comment
[15:0] WRID 2 Work Request ID. Unique identifier for every WQE
[31:16] Reserved 2
[95:32] LADDR 8 Local address. Data Buffer address for operations
[127:96] LENGTH 4 Length of the transfer
8’h00 -- RDMA WRITE
8’h01 -- RDMA_WRITE WITH IMMDT
8’h02 -- RDMA SEND
[135:128] OPCODE 1 8’h03 -- RDMA SEND WITH IMMDT
8’h04 -- RDMA READ
8’h0C -- RDMA SEND WITH INVALIDATE
All other values are reserved.
[159:136] Reserved 3 3
[223:160] ROFFSET 8 Remote offset address

ERNIC v4.0 13
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐1: WQE Structure (Cont’d)

Bitwidth Content Size (B) Comment
[255:224] RTAG 4 Remote TAG
RDMA SEND Data. If the data to be sent is less than or equal
[383:256] SDATA 16
to 16B, this field is used to represent the data.
[511:384] Reserved 16
[415:384] IMMDT DATA 4 Immediate data to be send in ImmDt header
[511:416] Reserved 12 Reserved

Work Completions
Work completions are posted for every WQE posted on the Send Queue. Completions are
not posted for Receive Queue (RQ) entries, instead doorbells are rung per Queue Pair (QP)
at the address pointed to by RQWPTRDBADDi when a new receive buffer is consumed by an
incoming RDMA SEND request. The structure of a Completion Queue Entry (CQE) is given in
Table 2-2. Each CQE is 4 bytes in size.

Table 2‐2: Completion Queue Entry Structure

Bitwidth Content Size (B) Comment
[15:0] WRID 2 Work Request ID. Unique identifier for every WQE
• 8'd0 = RDMA WRITE
• 8'd1 = RDMA WRITE Immediate
• 8'd2 = RDMA SEND
• 8'd3 = RDMA SEND Immediate
[23:16] OPCODE 1
• 8'd4 = RDMA READ
• 8'd5 - 8'd11 = RESERVED
• 8'd12 = RDMA SEND Invalidate
• 8'd13 - 8'd255 = RESERVED
[31:24] ERRFLAG 1 Error flag. Set to 8'b1 in case QP enters a fatal state

RDMA Queues
The ERNIC IP implements RDMA queues like Receive Queue (RQ), Send Queue (SQ), and
Completion Queue (CQ). These queues are referred to as Queue Pairs or QPs. SQ houses the
send WQEs posted by the user application.

ERNIC v4.0 14
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

X-Ref Target - Figure 2-2

SEND QUEUE RECEIVE QUEUE

stat_rq_pidb/
sq_pidb rq_wrptr_db

Pending WQEs Pending RQEs

sq_depth rq_depth

stat_cq_head/
rq_ci_db
sq_cmpl_db

WQEs processed but

awaiting response

stat_curr_sqptr_proc sq_ba rq_ba

COMPLETION QUEUE

sq_pidb

Pending WQEs

sq_depth

stat_cq_head/

cq_ba
X19878-102117

Figure 2‐2: RDMA Queues

The RQ houses the incoming RDMA SEND packets. The completion queue informs the user
application about the completed SEND WQEs. Each of these queues are implemented as
circular buffers. Various doorbell and write pointer registers define the current state of
these queues. Figure 2-2 shows the different queues and the variables/registers that define
the state of the queues. The highlighted variables are indirectly accessed by the ERNIC IP.
The register CQDBADDi points to the address for sq_cmpl_db and the RQWPTRDBADDi
register points to the address for the rq_wrptr_db.

These queues are implemented in memory regions outside the ERNIC IP. The ERNIC
accesses the IP through the various AXI master interfaces. See Table 3-2 for details of ERNIC
memory requirements.

The next few sections provide a brief overview of the incoming (RX) and outgoing (TX) data
flow of the ERNIC IP.

ERNIC v4.0 15
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

ERNIC RX Path
The ERNIC RX Path gets the packet data from the MAC through the AXI4-Stream interface.
All incoming packets are validated and all packet headers that fail packet validation are sent
to the error buffer (base address specified by ERRBUFBA[31:0]) if the value of
XRNICCONF[5] is set to 1. The header is prefixed with an error syndrome by the ERNIC RX
packet handler module as per Table 2-3. These buffers provide useful debug information for
incoming packet errors. RX path implements logic to detect ECN marked packets and logs
it in an interrupt status bit corresponding to that QP in CNPSCHDSTS*REG register. On
reception of ECN marked packet an interrupt is generated to notify driver. Driver generates
a CNP packet for that QP and schedule it through QP1 instead of QPN. Rx path implements
logic to detect incoming CNP packets and notifies QP manager. QP manager reduces the
outstanding on corresponding QP from 16 to 8. Further reception of CNPs will not have any
effect on the outgoing traffic.

Table 2‐3: Packet Validation Error Syndrome

Error
Syndrome Error related Precedence Description and impact Impact
to
Bit Index
MAC destination address received in the
0 MAC 4 Ethernet header does not match the ERNIC Packet dropped
MAC address configured
1 Reserved
2 IPv4/IPv6 5 IP Version not as per ERNIC configuration Packet dropped
3 IPv4 7 IPv4 header length not equal to 20 bytes Packet dropped
4 Reserved
5 IPv4 8 Flag bits in IPv4 are not 3'b010 Packet dropped
6 IPv4 8 Fragment Offset in IPv4 are not 0 Packet dropped
7 Reserved
IPv4 destination address in IPv4 header is not
8 IPv4/IPv6 8 Packet dropped
matching with ERNIC IPv4 address configured
9 IPv4 6 IPv4 header checksum error. Packet dropped
IPv4 Total Length field value is not with in
range.
10 IPv4/IPv6 8 Packet dropped
IPv6 Payload Length field value is not with in
range
11 Reserved
UDP Length field is not consistent with IPv4
12 UDP 9 Packet dropped
Total Length or IPv6 Payload Length
13 BTH 10 BTH Version is not 4'b0000 Packet dropped
QP Destination address in BTH is not
14 BTH 10 Packet dropped
supported

ERNIC v4.0 16
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐3: Packet Validation Error Syndrome (Cont’d)

Error Error related
Syndrome Precedence Description and impact Impact
Bit Index to

QP specified in the destination address of BTH

15 BTH 11 Packet dropped
is either not configured or not enabled
NAK-Invalid sent
16 BTH 14 Opcode sequence in not correct and QP moved to
FATAL state
NAk-Invalid sent
Unsupported or reserved opcode request is
17 BTH 10 and QP moved to
received from remote QP
FATAL state
For FIRST and MIDDLE request/response
18 BTH 14 Packet dropped
received BTH Pad bits are not 2'b00
Transport Transport Layer Payload is inconsistent with
19 14 Packet dropped
layer PMTU configured for that QP
Internal Request Queue is full and the packet is
20 13 RNR-NAK sent
error dropped
Transport NAK-Sequence
21 12 Incoming request PSN sequence is not correct
layer error
Transport
22 14 AETH Syndrome is malformed. Packet dropped
layer
Transport Bad response or NAK-Invalid response QP moved to FATAL
23 14
layer received state
24-26 Reserved
IP4(6) source address in IP4(6) header is not
27 IPv4/IPv6 5 matching with QP configured remote node Packet dropped
source address
MAC source address in the Ethernet header
28 MAC 4 does not match the QP configured remote Packet dropped
node MAC address
29 Reserved
30 Link layer 2 ICRC Error Packet dropped
31 Link layer 1 FCS Error Packet dropped

Most of the packet validation errors are handled entirely by hardware and no software
intervention is required. However, if an incoming packet causes the QP to enter into a FATAL
state, software intervention is required to process the error and to initiate a disconnection.
Such errors are available for the SW in the incoming packet error status buffers defined by
IPKTERRQBA, IPKTERRQSZ, and IPKTERRQWPTR registers. Each error status buffer entry is
64-bit wide. The format for the error status is as shown in Figure 2-3.

ERNIC v4.0 17
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

X-Ref Target - Figure 2-3

Figure 2‐3: Error Status Format

Fatal table decoding is shown in Table 2-4.

Table 2‐4: Decoding for FATAL Codes

FATAL Code Description Local/Remote Error
5'b00001 Opcode seq check fail Locally detected error
Request packet length is not as per PMTU configured
5'b00010 Locally detected error
OR Pad count check failed
Both opcode sequence error and packet length OR Pad
5'b00011 Locally detected error
count error occurred simultaneously
Unsupported request opcode received but with
5'b00100 Locally detected error
correct PSN
5'b00101 QP went into fatal due to WQE Processor Locally detected error
5’b00110 QP went into fatal due to response handler Locally detected error
5'b10010 Write packet length error or pad count check failed Locally detected error
5'b10001 Write opcode sequence check failed Locally detected error
5'b10011 Both the above errors occurred simultaneously Locally detected error
5'b10101 R-key check failed or access permission check failed Locally detected error
5'110110 RETH DMA length check failed Locally detected error
5'b10111 Both the above errors occurred simultaneously Locally detected error
5'b10000 Read request resources full Locally detected error
5'b01100 ACK response opcode is not correct Locally detected error
5'b01101 RNR-NAK down counter expired Locally detected error
5'b01010 NAK Invalid/RAE/ROE response received Remote Error
5'b11111 Read response last packet error Locally detected error
5'b11001 Bad AETH syndrome Locally detected error
5'b11110 Read response middle length check failed Locally detected error
5'b11100 Read response last instead of read resp only Locally detected error
5'b11011 Bad response error Locally detected error
5'b01001 RNR counter expired Locally detected error
5'b11010 Read Response only length check failed Locally detected error

Incoming RDMA SEND/WRITE/READ requests are expected on the RX side. All other types
of packets are response packets for the outgoing requests. The data flow for incoming
RDMA SEND requests is shown in Figure 2-4. The direction of arrows show the flow of data.
On receiving a valid RDMA SEND incoming packet on a connected QP, the packet content

ERNIC v4.0 18
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

(without the headers) is pushed into the RX Buffer for the relevant QP. The ERNIC rings the
RQ Producer Index Doorbell (RQPI DB), to indicate that a new packet is available, either
using the side band interface or through the AXI interface. This depends on the
configuration of QPCONFi[4]. An acknowledgment is also posted to the remote host at this
point. The user application may inform the ERNIC of having consumed the new packet by
ringing the RQ consumer Index Doorbell (RQCI DB). On receiving this doorbell, the
corresponding RX buffer is made available to be used for new incoming packets.
X-Ref Target - Figure 2-4

Initiator RNIC ERNIC User Application

RDMA SEND
packet sent
Packet posted on
RQ buffer, RQPli
doorbell rung on
sideband I/F,
STATRQPIDBi
updated

Explicit or
coalesced ACK sent

X19879-120220

Figure 2‐4: RDMA SEND RX Data Flow

For incoming RDMA READ/WRITE request support on any RDMA device and to send a read/
write request to other RDMA device in the network, the device should know the address,
length, and RKEY of the destination RDMA client. For this purpose, the application layer
should implement its own protocol to exchange the address and RKEY values before any
data transfers. The examples in Figure 2-5 and Figure 2-6 shows an additional RDMA_SEND
operation to exchange the required details for the RDMA READ/WRITE transaction. When
the details are exchanged, RNIC will send a RDMA READ/WRITE packet to ERNIC and ERNIC
will respond with ACK or RDMA read response. Figure 2-5 shows the flow of data for RDMA
READ transaction. Figure 2-6 shows the flow of data for RDMA WRITE transaction.

ERNIC v4.0 19
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

X-Ref Target - Figure 2-5

RNIC ERNIC User Application

WQE posted
RDMA SEND with
Capsule containing
address information
(WQE posted at the
target).

RDMA READ
Request

Data fetched
from the local
buffer

Single or
multiple RDMA
READ Response
packets
depending on
the DMA length.

X24862-120220

Figure 2‐5: RDMA READ Request RX Data Flow

ERNIC v4.0 20
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

X-Ref Target - Figure 2-6 pp

WQE posted
RDMA SEND with
Capsule containing
address information
(WQE posted at the
target).

RDMA WRITE
Request

Write Data written

into corresponding
memory

Explicit or
coalesced ACK sent

X24863-121020

Figure 2‐6: RDMA WRITE Request RX Data Flow

ERNIC TX Path
The TX Path consists of outgoing RDMA READ, RDMA WRITE transactions, and ACK packets
for incoming RDMA SEND/WRITE requests and responses for incoming RDMA READ
requests. Based on the SQPIi doorbell, the Send Work Queue requests are processed. The
DMA module is configured for data transfers for all outgoing transactions. The TX data flow
for RDMA WRITE/SEND is shown in Figure 2-7.

The user application requests the ERNIC IP to transmit an RDMA WRITE/SEND/READ packet
by posting a WQE on the SEND Queue for a particular QP and ringing the corresponding SQ
Producer Index Doorbell (SQPI DB). The ERNIC processes the WQE and pulls data for RDMA
WRITE/SEND commands based on the information provided in the WQE. This data along
with relevant headers is pushed out on the link. Once an acknowledgment is received from
the remote host, the ERNIC informs the user application of the successful completion of the
WQE by posting a CQE, based on the configuration of QPCONFi[5], and by posting the
completion count through the side band interface. The CQHEAD for the corresponding QP
is also updated. For RDMA READ requests, the RX packet handler intimates the TX path. For

ERNIC v4.0 21
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

this communication, an outstanding read request queue is available for each QP. The depth
of the queue is determined by a parameter for incoming request resources. When the
outstanding read request queue is full, it is indicated to the RX path. Any further requests to
the QP will result in an NAK-Invalid and the QP is moved to fatal state.

Note: The direction of arrows shows the flow of data.

X-Ref Target - Figure 2-7

RNIC ERNIC User Application

WQE posted,
SQPli DB rung

RDMA
SEND/WRITE
Packet(s) sent RDMA WRITE data
pulled by ERNIC from
Local buffer

CQ entry posted,
Completion count
sent through
Explicit or
sideband channel
coalesced ACK
CQHEADi updated

X19881-120220

Figure 2‐7: RDMA SEND/WRITE Data Flow

ERNIC v4.0 22
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Figure 2-8 shows the flow of RDMA READ requests to the remote host from the ERNIC.
X-Ref Target - Figure 2-8

RNIC ERNIC Target IP

WQE posted,
SQPli DB rung

RDMA READ
Packet sent

Read
Response(s)
received Read response pushed
to local buffer

CQ entry posted,
Completion count
sent through
sideband channel
CQHEADi updated
X19882-120220

Figure 2‐8: RDMA READ Data Flow

Figure 2-8 shows the flow for RDMA READ requests to the remote host from the ERNIC. The
user application requests the ERNIC IP to transmit an RDMA READ request to the remote
host by posting a WQE on the SEND Queue for a particular QP and ringing the
corresponding SQ Producer Index Doorbell (SQPI DB). ERNIC processes the WQE and
creates the request packet with relevant headers and pushes it out on the link. Once the
response data packets are received from the remote host, ERNIC writes the data (after
removing the headers) to the local buffer address provided in the WQE. It then informs the
user application of the successful completion of the WQE by posting a CQE, based on the
configuration of QPCONFi[5], and by posting the completion count through the side band
interface. The CQHEAD for the corresponding QP is also updated.

ERNIC v4.0 23
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Standards
This implementation is based on the standard and specifications described in InfiniBand
Architecture Specification Volume 1, Annex A16 RoCE and Annex 17 RoCE V2 [Ref 1].

Performance
The ERNIC IP is designed with an internal data path throughput of up to 100 Gb/s at a
frequency of 200 MHz.

For more details about resource utilization, see the Performance and Resource Utilization.

Resource Utilization
This section summarizes the estimated maximum performance for various modules within
the ERNIC IP. The data is separated into a table per device family. Each row describes a test
case. The columns are divided into test parameters and results. The test parameters include
the part information and the core-specific configuration parameters. Any configuration
parameters that are not listed have their default values; any parameters with a blank value
are disabled or set automatically by the IP core. Consult the product guide for this IP core
for a list of GUI parameter and user parameter mappings.

• Resource use numbers are taken from the utilization report issued at the end of an
implementation using the Out-of-Context flow in Vivado Design Suite.
• The Out-of-Context IP constraints include HD.CLK_SRC properties as required to ensure
correct hold timing closure. These properties are enabled using the following Tcl
command: set_param ips.includeClockLocationConstraints true
• The frequencies used for clock inputs are stated for each test case.
• LUT numbers do not include LUTs used as pack-thrus, but include LUTs used as
memory.
• Default Vivado® Design Suite 2018.1 settings are used. You can improve on these
numbers using different settings. However, because the surrounding circuitry will affect
placement and timing, these numbers might not repeat in a larger design.

For more details about resource utilization, see the Performance and Resource Utilization.

ERNIC v4.0 24
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Port Descriptions
Table 2-5 table describes the ports and their interface definitions

Table 2‐5: ERNIC IP Ports

Clock
Name I/O Width Domain Description

Clocking and Reset

m_axi_aclk I 1 AXI4 AXI4 clock
m_axi_aresetn I 1 AXI4 reset (active-Low)
s_axi_lite_aclk I 1 AXI4 AXI4-Lite clock
s_axi_lite_aresetn I 1 AXI4-Lite reset (active-Low)
system_resetn O 1 System reset
cmac_rx_clk I 1 RX user clock output from CMAC
cmac_rx_rst I 1 RX reset for user logic from CMAC
cmac_tx_clk I 1 TX user clock output from CMAC
cmac_tx_rst I 1 TX reset for user logic from CMAC
Response Handler AXI4 Master Interface
This interface is used by the response handler
to write completions in the completion queue
present in the DDR. See Appendix A of the
resp_hndler_m_axi_* I/O AXI4
Vivado Design Suite: AXI Reference Guide
(UG1037) [Ref 8] for more information on the
AXI4 signal.
RX Packet Handler AXI4 Master Interface to DDR
This interface is used by the RX packet handler
module to push the data from incoming MAD,
SEND, and RDMA WRITE packets to RQ buffer in
rx_pkt_hndler_ddr_m_axi* I/O AXI4 the DDR. Supports only 64B aligned
transaction. See Appendix A of the Vivado
Design Suite: AXI Reference Guide (UG1037)
[Ref 8] for more information on the AXI4 signal.
RX Packet Handler AXI4 Master Read Response Interface
This interface is used by the RX packet handler
to write the data received in read responses to
rx_pkt_hndler_rdrsp_m_axi* I/O AXI4 the DDR. See Appendix A of the Vivado Design
Suite: AXI Reference Guide (UG1037) [Ref 8] for
more information on the AXI4 signal.

ERNIC v4.0 25
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐5: ERNIC IP Ports (Cont’d)

Name I/O Width Clock Description

Domain
AXI4-Stream Slave Interface for Incoming RoCE Traffic
This interface provides RoCE packets from the
network interface to the IP. See Appendix A of
AXI4- the Vivado Design Suite: AXI Reference Guide
roce_cmac_s_ axis_* I/O
Stream (UG1037) [Ref 8] for more information on the
AXI4 signal.

AXI4-Stream Slave Interface for Incoming Non-RoCE Traffic

This interface provides non-RoCE packets from
the network interface to the IP. See Appendix A
AXI4-
non_roce_cmac_s_ axis_* I/O of the Vivado Design Suite: AXI Reference Guide
Stream
(UG1037) [Ref 8] for more information on the
AXI4 signal.
Non-RoCE AXI4-Stream Interface from DMA Module
Incoming non-RDMA path from DMA module
AXI4- to IP. See Appendix A of the Vivado Design
non_roce_dma_s_axis_* I/O
Stream Suite: AXI Reference Guide (UG1037) [Ref 8] for
more information on the AXI4 signal.
Non-RoCE AXI4-Stream Interface to DMA Module
AXI4 Outgoing non-RDMA path from IP to DMA
AXI4- module. See Appendix A of the Vivado Design
non_roce_dma_m_axis_* I/O
Stream Suite: AXI Reference Guide (UG1037) [Ref 8] for
more information on the AXI4 signal.
CMAC AXI4-Stream Interface
cmac_tx This interface is used to send out RoCE and
cmac_m_axis_* I/O
_clk non-RoCE packets from ERNIC to CMAC.
WQE Processor AXI Master Interface
This interface is used by the WQE processor
engine to read the data sent in outgoing RDMA
SEND/WRITE and READ responses from the
wqe_proc_top_m_axi_* I/O AXI4
local buffer. See Appendix A of the Vivado
Design Suite: AXI Reference Guide (UG1037)
[Ref 8] for more information on the AXI4 signal.
WQE Processor AXI4 Master Interface to DDR
This interface is used by the WQE processor to
write the data of outgoing write packets in the
wqe_proc_wr_ddr_m_axi_* I/O AXI4 write retry buffer. See Appendix A of the Vivado
Design Suite: AXI Reference Guide (UG1037)
[Ref 8] for more information on the AXI4 signal.

ERNIC v4.0 26
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐5: ERNIC IP Ports (Cont’d)

Name I/O Width Clock Description

Domain
AXI4-Lite Slave Interface for Register Programming
This interface is used by the processor to
configure the ERNIC registers. See Appendix A
AXI4-
s_axi_lite_* I/O of the Vivado Design Suite: AXI Reference Guide
Lite
(UG1037) [Ref 8] for more information on the
AXI4 signal.
QP Manager AXI4 Master Interface
This interface is used by the QP manager to
fetch the send queue WQEs from the DDR. See
qp_mgr_m_axi_* I/O AXI4 Appendix A of the Vivado Design Suite: AXI
Reference Guide (UG1037) [Ref 8] for more
information on the AXI4 signal.
Invalidate or Immediate Data AXI4-Stream Interface
This interface is used to provide IETH or Immdt
headers along with 32-bit additional data to
ieth_immdt_m_axis_* I/O AXI4S
external hardware logic. The use of this
information is left to user application.
HW Handshake Ports for RQ Doorbells
RDMA-SEND Producer Index Doorbell Value
rx_pkt_hndler_o_rq_db_data O 32 AXI4
from ERNIC
RDMA-SEND Producer Index Doorbell Address
rx_pkt_hndler_o_rq_db_addr O 13 AXI4
(4 Bytes per QP; for 127 QPs)
RDMA-SEND Producer Index Doorbell Valid.
When this signal is asserted High,
rx_pkt_hndler_o_rq_db_addr and
rx_pkt_hndler_o_rq_db_data are valid.
rx_pkt_hndler_o_rq_db_data_valid O 1 AXI4 Until rx_pkt_hndler_i_rq_db_rdy is not sampled
High, this signal remains asserted and,
rx_pkt_hndler_o_rq_db_addr and
rx_pkt_hndler_o_rq_db_data signals will hold
the same values.
Ready from the target signaling data and
rx_pkt_hndler_i_rq_db_rdy I 1 AXI4
address are accepted
HW Handshake Ports for CQ Doorbells
Send WQE Completion queue Doorbell count
resp_hndler_o_send_cq_db_cnt O 32 AXI4
from ERNIC
Send WQE Completion queue Doorbell address
resp_hndler_o_send_cq_db_addr O 13 AXI4
(4 Bytes per QP; for 127 QPs)

ERNIC v4.0 27
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐5: ERNIC IP Ports (Cont’d)

Name I/O Width Clock Description

Domain
Send WQE Completion Doorbell Valid. When
this signal is asserted High,
resp_hndler_o_send_cq_db_addr and
resp_hndler_o_send_cq_db_cnt are valid. Until
resp_hndler_o_send_cq_db_cnt_valid O 1 AXI4 resp_hndler_i_send_cq_db_rdy is not sampled
High, this signal remains asserted and,
resp_hndler_o_send_cq_db_addr and
resp_hndler_o_send_cq_db_cnt signals will hold
the same values.
Send WQE Completion Doorbell Ready. Ready
signal should go High when the target
resp_hndler_i_send_cq_db_rdy I 1 AXI4
application accepts the current doorbell
transaction.
HW Handshake Ports for SQ PI Doorbells
Send WQE Producer Index Doorbell Value from
i_qp_sq_pidb_hndshk I 16 AXI4
target application
i_qp_sq_pidb_wr_addr_hndshk I 32 AXI4 Send WQE Producer Index Doorbell Address
Send WQE Producer Index Doorbell Valid. After
the target application posts WQE(s), it should
assert this signal High with valid
i_qp_sq_pidb_wr_addr_hndshk and
i_qp_sq_pidb_wr_valid_hndshk I 1 AXI4 i_qp_sq_pidb_hndshk. Target application
should keep this signal asserted and hold
i_qp_sq_pidb_wr_addr_hndshk and
i_qp_sq_pidb_hndshk signals until it samples
o_qp_sq_pidb_wr_rdy as High.
Send WQE Producer Index Doorbell Ready.
o_qp_sq_pidb_wr_rdy O 1 AXI4 Ready signal asserted High when ERNIC
accepts the Doorbell value.
HW Handshake Ports for RQ CI Doorbells
RDMA-SEND Consumer Index Doorbell value
i_qp_rq_cidb_hndshk I 16 AXI4
from target application
RDMA-SEND Consumer Index Doorbell register
i_qp_rq_cidb_wr_addr_hndshk I 32 AXI4
address
RDMA-SEND Consumer Index Doorbell Valid.
After target application processes incoming
RDMA-SEND command(s), it should assert this
signal High with valid
i_qp_rq_cidb_wr_addr_hndshk and
i_qp_rq_cidb_wr_valid_hndshk I 1 AXI4
i_qp_rq_cidb_hndshk. Target application should
keep this signal asserted and hold
i_qp_rq_cidb_wr_addr_hndshk and
i_qp_rq_cidb_hndshk signals until it samples
o_qp_rq_cidb_wr_rdy as High.
RDMA SEND Consumer Index Doorbell value is
o_qp_rq_cidb_wr_rdy O 1 AXI4
accepted by ERNIC

ERNIC v4.0 28
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐5: ERNIC IP Ports (Cont’d)

Name I/O Width Clock Description

Domain
Priority Flow Control Ports
cmac_rx Pause request signal. This signal gets asserted
stat_rx_pause_req[8:0] I 9
_clk by CMAC IP for valid quanta period.
This bus gets asserted when buffer thresholds
cmac_tx
ctl_tx_pause_req[8:0] O 9 are between XON and XOFF. priority bit is set
_clk
through priority register.
cmac_tx This signal is hardwired to 0 and can be
ctl_tx_resend_pause O 1
_clk connected to CMAC IP signal.
Interrupts
This bit is set when any one of the interrupts in
rnic_intr O 1 AXI4
the register INTEN occurs.
Debug Counter Enabling Signals
o_global_dbg_cnt_en O 1 AXI4 Enables debug signals
o_global_dbg_cnt_clr O 1 AXI4 Clears the debug counters

Parameter Descriptions
As many features in the ERNIC controller design are controlled using parameters, the
controller implementation can be uniquely tailored using only the resources required for
the desired functionality. This approach also achieves the best possible performance with
the lowest resource usage.

Table 2-6 lists the parameter descriptions and default values.

Table 2‐6: ERNIC Parameters

Parameter Name Description Default

Value
Maximum number of Queue Pairs. The minimum number of
C_NUM_QP 8
queue pairs is 8 and the maximum number is 2048
C_M_AXI_ADDR_WIDTH AXI Master address width (supported values are 32 and 64) 32
C_M_AXI_ID_WIDTH AXI Master ID width 1
C_EN_INITIATOR_LITE Enable incoming RDMA READ and RDMA WRITE packets 0
C_EN_DEBUG_PORTS Enabling the debug ports for debug purposes 0

ERNIC v4.0 29
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Register Space
All the ERNIC registers are synchronous to the AXI4-Lite domain. Any bits not specified in
register tables below are considered reserved and return the values as 0 upon read. The
power-on reset values of control registers are 0 unless specified in the definition. You
should always write the reserved locations with a 0 unless stated otherwise. Only address
offsets are listed in the table below and the base address is configured by the AXI
interconnect at system level. The contents of the memory region table are used for header
validation as shown in the following figure.
X-Ref Target - Figure 2-9

PD Number from Per QP Configuation

RSVD PD Number
31 23
Step 1 Step 2
Match PD
Number
RETH
Use MR Index which is part PD Information
of RETH Rkey
Memory Region Table 31 23 0
to fetch PD information
from PD table RSVD PD Number
31 0
Virtual Address LSB
RETH Virtual Address LSB Virtual Address MSB
RETH Virtual Address MSB Physical Addr LSB
MR index Physical Addr MSB
RETH DMA Length Reserved R_Key
31 7 0 DMA Length
DMA Length MSB RSVD Access

31 16 7 0

Match
R_KEY

Buffer
Range
Validation

Check

X21690-121020

Figure 2‐9: MR Table Implementation

ERNIC v4.0 30
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details

Address Access Register Name Details
0x00 + ((i)
Protection Domain Number from the memory region table
x 0x0100) Protection Domain
RW Width = 24
Where i=0 Number (PDPDNUM)
Default value = 0
to 2047
This register provides the Virtual Add of the buffer base address
0x04 + ((i) Virtual address LSB LSB.
RW
x 0x0100) (VIRTADDRLSB) Width = 32
Default value = 0
This register provides the Virtual Add of the buffer base address
0x08 + ((i) Virtual address MSB MSB.
RW
x 0x0100) (VIRTADDRMSB) Width = 32
Default value = 0
This register provides the Write/Read buffer base address MSB.
0x0C + ((i) Buffer Base address LSB
Width = 32
x 0x0100) (BUFBASEADDRLSB)
Default value = 0
This register provides the Write/Read buffer base address LSB.
0x10 + ((i) Buffer base address MSB
RW Width = 32
x 0x0100) (BUFBASEADDRMSB)
Default value = 0
This register provides the Write/Read buffer R_KEY register.
0x14 + ((i) Buffer R_Key
RW Width = 8
x 0x0100) (BUFRKEY)
Default value = 0
This register provides the Write/Read buffer length LSB register.
0x18 + ((i) Write/Read buffer length
RW Width = 32
x 0x0100) (WRRDBUFLEN)
Default value = 0
This register provides [7:0] the Access description of the
Protection Domain and [31:16] Write/Read buffer length MSB
register. Default value is 0.
• [3:0]: Access type
° 4'b0000: READ Only
0x1C + ((i) Access Description
RW ° 4'b0001: Write Only
x 0x0100) (ACCESSDESC)
° 4'b0010: Read and Write
° Other values: Not supported
• [15:4]: Reserved
• [31:16]: w_r_buf_len_msb. Register provides the Write/Read
buffer length [47:32]

ERNIC v4.0 31
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register provides the basic global (not QP specific)
configuration controls for the ERNIC IP
• [0]: ERNIC Enable
• [1]: Reserved
• [2]: Reserved
• [4:3]: TX ACK generation.
° 00 – (default) ACK only generated on explicit ACK request in
XRNIC Configuration
0x10_0000 RW the incoming packet or on timeout
(XRNICCONF)
° 01 – ACK generated only on timeout
° 10 – ACK generated only on explicit ACK request in the
incoming packet
• [5]: Error buffer enable. This bit is set to enable the error buffer
• [7:6]: Reserved
• [23:8]: UDP source port for outgoing packets
• [31:24]: Reserved
This register provides advanced configuration controls for the
ERNIC IP.
• [0]: SW override enable. Allows SW write access to the following
Read Only Registers – CQHEADn, STATCURRSQPTRn, and
STATRQPIDBn (where is the QP number)
• [1]: Reserved
• [2]: retry_cnt_fatal_dis
• [15:3]: Reserved
XRNIC Advance
0x10_0004 RW Configuration • [19:16]: Base count width
(XRNICADCONF) ° Approximate number of system clocks that make 4096us.
° For 400 MHz clock -->Program decimal 11
° For 200 MHz clock --> Program decimal 10
° For 125 MHz clock --> Program decimal 09
° For 100 MHz clock --> Program decimal 09
° For N MHz clock ---> Value should be CLOG2(4.096 *N)
• [30:20] Software Override QP Number
• [31]: Reserved
This register provides XON and XOFF Configuration for RoCE
Traffic.
XRNIC_BUF_
0x10_0008 RW • [15:0]: XON Value for RoCE
THRESHOLD_ROCE
• [31:16]: XOFF Value for RoCE
Default value = 0x0096005A
This register provides PFC configuration.
• [0]: Pause enable RoCE
• [1]: Pause enable non-RoCE
0x10_000C RW XRNIC_PAUSE_CONF • [7:4]: Pause priority for RoCE (Binary encoding)
• [11:8]: Pause priority for non-RoCE (Binary encoding)
• [13]: Disable priority check (applicable only for TX data path)
Default value = 0x800

ERNIC v4.0 32
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This is the MAC Address for local (ERNIC) device and should be
MAC XRNIC Address LSB
0x10_0010 RW configured before the XRNICCONF[0] is set to 1.
(MACXADDLSB)
• [31:0]: MAC local address LSB
MAC Address for local (ERNIC) device. Should be configured
MAC XRNIC Address MSB before the XRNICCONF[0] is set to 1.
0x10_0014 RW
(MACXADDMSB) • [15:0]: MAC local address MSB
• [31:16]: Reserved
This register provides XON and XOFF configuration for non-RoCE
XRNIC_BUF_ traffic.
0x10_0018 RW THRESHOLD_NON_ • [15:0]: XON value for non-RoCE
ROCE • [31:16]: XOFF value for non-RoCE
Default value = 0x0096005A
IPv6 address [32:0] for local (ERNIC) device. Should be configured
IPv6 Address 1
0x10_0020 RW before the XRNICCONF[0] is set to 1.
(IPv6XADD1)
[31:0]: IP address [31:0]
IPv6 address [63:32] for local (ERNIC) device. Should be
IPv6 Address 2
0x10_0024 RW configured before the XRNICCONF[0] is set to 1.
(IPv6XADD2)
[31:0]: IP address [63:32]
IPv6 address [95:64] for local (ERNIC) device. Should be
IPv6 Address 3
0x10_0028 RW configured before the XRNICCONF[0] is set to 1.
(IPv6XADD3)
[31:0]: IP address [95:64]
IPv6 address [127:96] for local (ERNIC) device. Should be
IPv6 Address 4
0x10_002C RW configured before the XRNICCONF[0] is set to 1.
(IPv6XADD4)
[31:0]: IP address [127:96]
This register provides the base address for Error buffers. The
ERNIC IP updates these buffers with incoming packets that fail
validation. The writes to this buffer for all validation errors are
Error Buffer Base Address
0x10_0060 RW enabled by writing a 1 to XRNICCONF[5]. If this bit is disabled, only
(ERRBUFBA)
packets that cause the QP to move to a FATAL state are written to
the error buffer.
• [31:0]: Error buffer base address LSB 32 bits
This register contains the information about how many QPs are
going to be active.
0x10_0044 RW XRNIC_CONF_QP_EN
• [10:0]: Number of QPs enabled
• [31:11]: Reserved
Error Buffer Base Address This register provides the msb base address for Error buffers.
0x10_0064 RW
msb (ERRBUFBAMSB) • [31:0]: Error buffer base address MSB 32 bits
This register provides the size of the error buffer
Error Buffer Size
0x10_0068 RW • [15:0]: Number of Error buffers
(ERRBUFSZ)
• [31:16]: Size of each Error buffer
This register is updated by the ERNIC IP when any error packet is
Error Buffer write pointer written to the error buffer. The pointer informs the driver of the
0x10_006C RO
(ERRBUFWPTR) number of error packets received.
• [15:0]: Write pointer doorbell for Error Buffer

ERNIC v4.0 33
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
IPv4 XRNIC Address IPv4 address for local (ERNIC) device.
0x10_0070 RW
(IPv4XADD) • [31:0]: IPv4 address
This register is used to configure the base address for outgoing
Outgoing Pkt Error status
packet error status queue. See, ERNIC RX Path for details of the
0x10_0078 RW Queue Base address
outgoing packet error status queue.
(OPKTERRQBA)
[31:0]: Outgoing Pkt Error Status Queue base address
This register is used to configure the msb base address for
Outgoing Pkt Error status
outgoing packet error status queue. See, ERNIC RX Path for details
0x10_007C RW Queue Base address
of the outgoing packet error status queue.
(OPKTERRQBAMSB)
[63:32]: Outgoing Pkt Error Status Queue base address msb
Outgoing Error status Q
0x10_0080 RW [15:0]: Number of entries in outgoing error Q.
size (OUTERRSTSQSZ)
Outgoing Error status Q Outgoing Error status Q write pointer doorbell. The hardware
0x10_0084 RW write pointer Doorbell keeps writing to the buffer in a circular fashion. Does not take care
(OPTERRSTSQQPTRDB) of buffer overflow. Needs to be handled in SW.
This register is used to configure the base address for incoming
Incoming Pkt Error Status
packet error status queue. See, ERNIC RX Path for details of the
0x10_0088 RW Queue Base address
incoming packet error status queue.
(IPKTERRQBA)
[31:0]: Incoming Pkt Error Status Queue base address
Incoming Pkt Error Status
This register is used to configure the msb base address for
0x10_008C RW Queue Base address MSB
incoming packet error status queue.
(IPKTERRQBAMSB)
This register is used to configure the size of the incoming packet
Incoming Pkt Error Status error status queue.
0x10_0090 RW
Queue Size (IPKTERRQSZ) • [15:0]: Number of incoming error pkt status queue entries
• [31:16]: Reserved
This status register provides information on the number of
Incoming Pkt Error Status incoming error packets received by the ERNIC IP.
0x10_0094 RO write pointer [15:0]: Write pointer doorbell for incoming error status queue.
(IPKTERRQWPTR) ERNIC IP writes to the queue in a circular manner without taking
care of overflow. This needs to be handled in SW.
These data buffers are used by the ERNIC to save all outgoing
RDMA WRITE data until it is acknowledged by the remote host. In
Data Buffer Base Address
0x10_00A0 RW the event of retransmission, the retried data is pulled from these
(DATBUFBA)
buffers.
[31:0]: Data Buffer base address
These data buffers are used by the ERNIC to save all outgoing
Data Buffer Base RDMA WRITE data until it is acknowledged by the remote host. In
0x10_00A4 RW Address MSB the event of retransmission, the retried data is pulled from these
(DATBUFBAMSB) buffers.
[63:32]: Data Buffer base address MSB
This register is used to configure the size of the data buffers.
Data Buffer Size
0x10_00A8 RW • [15:0]: Number of data buffers
(DATBUFSZ)
• [31:16]: Data buffer size in bytes

ERNIC v4.0 34
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register is used to ring the doorbell when hardware
handshake is just enabled.
• [15:0]: Reserved
Connect IP configuration • [26:16]: Queue pair ID
0x10_00AC RW
(CON_IO_CONF) • [30:27]: Reserved
• [31]: Connect IO ready indication. This bit is read only and when
set to 1'b1 indicates that the connect IO configuration is done
and ready to accept for another QPID
These error buffers are used by the ERNIC to save all error
Response Error pkt buffer
response error packet header information. This register holds the
0x10_00B0 RW base address
LSB 32 bits of the response error packet buffer
(RESPERRPKTBA)
[31:0]: Response Error pkt Buffer base address lsb
Response Error pkt buffer
This register holds the MSB 32 bits of the response error buffer
0x10_00B4 RW base address msb
[31:0]: Response Error pkt Buffer base address msb
(RESPERRPKTBAMSB)
This register provides information of size of each entry and depth
of response error buffer
Response Error pkt buffer [15:0]: Number of entries for response error buffer
0x10_00B8 RW
size address (RESPERRSZ) [31:16]: Size of each entry of response error buffer
Note: This register should be programmed to a valid value for proper
operation.

This register indicate current write pointer of the response error

Response Error pkt buffer buffer
0x10_00BC RW write pointer
[15:0]: Current write pointer
(RESPERRWRPTR)
[31:16]: Reserved
Global Status Registers
This is a status register which provides information about
Incoming SEND/Read incoming RDMA SEND and RDMA READ response packets.
0x10_0100 RO Response Pkt count
• [15:0]: Incoming SEND packet count
(INSRRPKTCNT)
• [31:16]: Incoming Read Response packet count
This is a status register which provides information about
incoming acknowledgment and Management Datagram (MAD)
Incoming ACK/MAD Pkt packets.
0x10_0104 RO
count (INAMPKTCNT)
• [15:0]: Incoming ACK packet count
• [31:16]: Incoming MAD packet count
This is a status register which provides information about
Outgoing IO (SEND/ outgoing RDMA SEND and RDMA READ/WRITE packets.
0x10_0108 RO READ/WRITE) Pkt count
• [15:0]: Outgoing SEND packet count
(OUTIOPKTCNT)
• [31:16]: Outgoing RDMA READ/WRITE packet count
This is a status register which provides information about
Outgoing ACK/MAD Pkt outgoing acknowledgment and MAD packets.
0x10_010C RO
count (OUTAMPKTCNT) • [15:0]: Outgoing ACK packet count
• [31:16]: Outgoing MAD packet count

ERNIC v4.0 35
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This status register provides details of the last incoming packet
received.
Last incoming pkt • [7:0]: Opcode of the last incoming packet
0x10_0110 RO
(LSTINPKT) • [19:8]: PSN LSB bits of the last incoming packet
• [30:20]: QPID of the last incoming packet
• [31]: Reserved
This status register provides the details of the last outgoing
packet.
Last outgoing pkt • [7:0]: Opcode of the last outgoing packet
0x10_0114 RO
(LSTOUTPKT) • [19:8]: PSN LSB bits of the last outgoing packet
• [30:20]: QPID of the last outgoing packet
• [31]: Reserved
This status register provides information on incoming invalid or
Incoming Invalid/ duplicate packets.
0x10_0118 RO Duplicate pkt count
• [15:0]: Incoming Invalid packet count
(ININVDUPCNT)
• [31:16]: Incoming Duplicate packet count
This status register provides the status of WQE processor engine
and is used for debug purposes.
• [2:0]: WQE processor state
• [3]: reserved
• [4]: retry buffers unavailable
WQE Processor Status • [5]: internal FIFO empty status
0x10_0124 RO
(WQEPROCSTS) • [6]: Buffer manager busy signal
• [11:7]: reserved
• [12]: Back pressure indication from CMAC. Sticky bit
• [15:12]: Header buffer manager FSM state
• [23:16]: Buffer tail pointer
• [31:24]: Buffer head pointer
This status register provides the status of QP manager and is used
for debug purposes.
QP Manager Status • [0]: WQE Cache full
0x10_012C RO
(QPMSTS) • [1]: WQE Cache empty
• [15:2]: Reserved
• [31:16]: Count of WQEs processed
This status register is provides details of all incoming and dropped
Incoming all/dropped pkt packets.
0x10_0130 RO
count (INALLDRPPKTCNT) • [15:0]: Incoming dropped packet count
• [31:16]: All incoming packet count
This status register is provides details of all incoming NAK packets.
Incoming NAK pkt count
0x10_0134 RO • [15:0]: Incoming NAK packet count
(INNAKPKTCNT)
• [31:16]: Reserved
This status register is provides details of all outgoing NAK packets.
Outgoing NAK pkt count
0x10_0138 RO • [15:0]: Outgoing NAK packet count
(OUTNAKPKTCNT)
• [31:16]: Reserved

ERNIC v4.0 36
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This status register provides the status of Response Handler
module and is used for debug purposes.
Response handler Status • [10:0]: Arbitrated QP Index
0x10_013C RO
(RESPHNDSTS) • [16:11]: Response handler FSM state
• [17]: Acks to process
• [31:18]: Reserved
Retry count status This status register provides details of retransmitted packets.
0x10_0140 RO
(RETRYCNTSTS) [31:0] Retry count
Incoming CNP packet
0x10_0174 RO Incoming CNP packet count
count (INCNPPKTCNT)
Outgoing CNP packet
0x10_0178 RO Outgoing CNP packet count
count (OUTCNPPKTCNT)
Outgoing read response
0x10_017C RO packet count Outgoing read response packet count
(OUTRDRSPPKTCNT)
This register provides the configuration control for interrupts
generated by the ERNIC IP.
• [0]: Incoming packet validation error interrupt enable
• [1]: Incoming MAD packet received interrupt enable
• [2]: Reserved
• [3]: RNR NACK generated interrupt enable
0x10_0180 RW Interrupt Enable (INTEN) • [4]: WQE completion interrupt enable (asserted for QPs for
which QPCONF[3] bit is set)
• [5]: Illegal opcode posted in SEND Queue interrupt enable
• [6]: RQ Packet received interrupt enable (asserted for QPs for
which QPCONF[2] bit is set
• [7]: Fatal error received interrupt enable
• [8]: Interrupt enable for CNP scheduling
This status register provides the cause of an asserted interrupt.
• [0]: Incoming packet validation error interrupt
• [1]: Incoming MAD packet received interrupt
• [2]: Reserved
• [3]: RNR NACK generated interrupt
• [4]: WQE completion interrupt (asserted for QPs for which
0x10_0184 W1C Interrupt Status (INTSTS)
QPCONF[3] bit is set)
• [5]: Illegal opcode posted in SEND Queue interrupt
• [6]: RQ Packet received interrupt (asserted for QPs for which
QPCONF[2] bit is set)
• [7]: Fatal error received interrupt
• [8]: Interrupt enable for CNP scheduling
RCV Q interrupt status This register provides the RQ interrupt status for QPs 0 to 31. Bit
0x10_0190 WC1 (bitwise) QP 0-31 [i] provides the interrupt status for RQi where i=0 to 31.
(RQINTSTS1) [i]: Interrupt status for RQ i

ERNIC v4.0 37
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
RCV Q interrupt status This register provides the RQ interrupt status for QPs 32 to 63. Bit
0x10_0194 WC1 (bitwise) QP 32-63 [i] provides the interrupt status for RQ(i+32) where i=0 to 31.
(RQINTSTS2) [i]: Interrupt status for RQ(i+32)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 64 to 95. Bit
0x10_0198 WC1 (bitwise) QP 64-95 [i] provides the interrupt status for RQ(i+64) where i=0 to 31.
(RQINTSTS3) [i]: Interrupt status for RQ(i+64)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 96 to 127. Bit
0x10_019C WC1 (bitwise) QP 96-127 [i] provides the interrupt status for RQ(i+96) where i=0 to 31.
(RQINTSTS4) [i]: Interrupt status for RQ(i+96)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 128 to 159.
0x10_01A0 WC1 (bitwise) QP 128-159 Bit [i] provides the interrupt status for RQ(i+128) where i=0 to 31.
(RQINTSTS5) [i]: Interrupt status for RQ(i+128)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 160 to 191.
0x10_01A4 WC1 (bitwise) QP 160-191 Bit [i] provides the interrupt status for RQ(i+160) where i=0 to 31.
(RQINTSTS6) [i]: Interrupt status for RQ(i+160)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 192 to 223.
0x10_01A8 WC1 (bitwise) QP 192-223 Bit [i] provides the interrupt status for RQ(i+192) where i=0 to 31.
(RQINTSTS7) [i]: Interrupt status for RQ(i+192)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 224 to 255.
0x10_01AC WC1 (bitwise) QP 224-255 Bit [i] provides the interrupt status for RQ(i+224) where i=0 to 31.
(RQINTSTS8) [i]: Interrupt status for RQ(i+224)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 256 to 287.
0x10_01B0 WC1 (bitwise) QP 256-287 Bit [i] provides the interrupt status for RQ(i+256) where i=0 to
(RQINTSTS9) 31.[i]: Interrupt status for RQ(i+256)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 288 to 319.
0x10_01B4 WC1 (bitwise) QP 288-319 Bit [i] provides the interrupt status for RQ(i+288) where i=0 to
(RQINTSTS10) 31.[i]: Interrupt status for RQ(i+288)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 320 to 351.
0x10_01B8 WC1 (bitwise) QP 320-351 Bit [i] provides the interrupt status for RQ(i+320) where i=0 to
(RQINTSTS11) 31.[i]: Interrupt status for RQ(i+320)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 352 to 383.
0x10_01BC WC1 (bitwise) QP 352-383 Bit [i] provides the interrupt status for RQ(i+352) where i=0 to
(RQINTSTS12) 31.[i]: Interrupt status for RQ(i+352)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 384 to 415.
0x10_01C0 WC1 (bitwise) QP 384-415 Bit [i] provides the interrupt status for RQ(i+384) where i=0 to
(RQINTSTS13) 31.[i]: Interrupt status for RQ(i+384)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 416 to 447.
0x10_01C4 WC1 (bitwise) QP 416-447 Bit [i] provides the interrupt status for RQ(i+416) where i=0 to
(RQINTSTS14) 31.[i]: Interrupt status for RQ(i+416)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 448 to 479.
0x10_01C8 WC1 (bitwise) QP 448-479 Bit [i] provides the interrupt status for RQ(i+448) where i=0 to
(RQINTSTS15) 31.[i]: Interrupt status for RQ(i+448)

ERNIC v4.0 38
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
RCV Q interrupt status This register provides the RQ interrupt status for QPs 480 to 511.
0x10_01CC WC1 (bitwise) QP 480-511 Bit [i] provides the interrupt status for RQ(i+480) where i=0 to
(RQINTSTS16) 31.[i]: Interrupt status for RQ(i+480)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 512 to 543.
0x10_01D0 WC1 (bitwise) QP 512-543 Bit [i] provides the interrupt status for RQ(i+512) where i=0 to
(RQINTSTS17) 31.[i]: Interrupt status for RQ(i+512)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 544 to 575.
0x10_01D4 WC1 (bitwise) QP 544-575 Bit [i] provides the interrupt status for RQ(i+544) where i=0 to
(RQINTSTS18) 31.[i]: Interrupt status for RQ(i+544)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 576 to 607.
0x10_01D8 WC1 (bitwise) QP 576-607 Bit [i] provides the interrupt status for RQ(i+576) where i=0 to
(RQINTSTS19) 31.[i]: Interrupt status for RQ(i+576)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 608 to 639.
0x10_01DC WC1 (bitwise) QP 608-639 Bit [i] provides the interrupt status for RQ(i+608) where i=0 to
(RQINTSTS20) 31.[i]: Interrupt status for RQ(i+608)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 640 to 671.
0x10_01E0 WC1 (bitwise) QP 640-671 Bit [i] provides the interrupt status for RQ(i+640) where i=0 to
(RQINTSTS21) 31.[i]: Interrupt status for RQ(i+640)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 672 to 703.
0x10_01E4 WC1 (bitwise) QP 672-703 Bit [i] provides the interrupt status for RQ(i+672) where i=0 to
(RQINTSTS22) 31.[i]: Interrupt status for RQ(i+672)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 704 to 735.
0x10_01E8 WC1 (bitwise) QP 704-735 Bit [i] provides the interrupt status for RQ(i+704) where i=0 to
(RQINTSTS23) 31.[i]: Interrupt status for RQ(i+704)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 736 to 767.
0x10_01EC WC1 (bitwise) QP 736-767 Bit [i] provides the interrupt status for RQ(i+736) where i=0 to
(RQINTSTS24) 31.[i]: Interrupt status for RQ(i+736)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 768 to 799.
0x10_01F0 WC1 (bitwise) QP 768-799 Bit [i] provides the interrupt status for RQ(i+768) where i=0 to
(RQINTSTS25) 31.[i]: Interrupt status for RQ(i+768)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 800 to 831.
0x10_01F4 WC1 (bitwise) QP 800-831 Bit [i] provides the interrupt status for RQ(i+800) where i=0 to
(RQINTSTS26) 31.[i]: Interrupt status for RQ(i+800)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 832 to 863.
0x10_01F8 WC1 (bitwise) QP 832-863 Bit [i] provides the interrupt status for RQ(i+832) where i=0 to
(RQINTSTS27) 31.[i]: Interrupt status for RQ(i+832)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 864 to 895.
0x10_01FC WC1 (bitwise) QP 864-895 Bit [i] provides the interrupt status for RQ(i+864) where i=0 to
(RQINTSTS28) 31.[i]: Interrupt status for RQ(i+864)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 896 to 927.
0x10_0200 WC1 (bitwise) QP 896-927 Bit [i] provides the interrupt status for RQ(i+896) where i=0 to
(RQINTSTS29) 31.[i]: Interrupt status for RQ(i+896)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 928 to 949.
0x10_0204 WC1 (bitwise) QP 928-949 Bit [i] provides the interrupt status for RQ(i+928) where i=0 to
(RQINTSTS30) 31.[i]: Interrupt status for RQ(i+928)

ERNIC v4.0 39
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
RCV Q interrupt status This register provides the RQ interrupt status for QPs 960 to 991.
0x10_0208 WC1 (bitwise) QP 960-991 Bit [i] provides the interrupt status for RQ(i+960) where i=0 to
(RQINTSTS31) 31.[i]: Interrupt status for RQ(i+960)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 992 to 1023.
0x10_020C WC1 (bitwise) QP 992-1023 Bit [i] provides the interrupt status for RQ(i+992) where i=0 to
(RQINTSTS32) 31.[i]: Interrupt status for RQ(i+992)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1024 to
0x10_0210 WC1 (bitwise) QP 1024-1055 1055. Bit [i] provides the interrupt status for RQ(i+1024) where i=0
(RQINTSTS33) to 31.[i]: Interrupt status for RQ(i+1024)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1056 to
0x10_0214 WC1 (bitwise) QP 1056-1087 1087. Bit [i] provides the interrupt status for RQ(i+1056) where i=0
(RQINTSTS34) to 31.[i]: Interrupt status for RQ(i+1056)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1088 to
0x10_0218 WC1 (bitwise) QP 1088-1119 1119. Bit [i] provides the interrupt status for RQ(i+1088) where i=0
(RQINTSTS35) to 31.[i]: Interrupt status for RQ(i+1088)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1120 to
0x10_021C WC1 (bitwise) QP 1120-1151 1151. Bit [i] provides the interrupt status for RQ(i+1120) where i=0
(RQINTSTS36) to 31.[i]: Interrupt status for RQ(i+1120)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1152 to
0x10_0220 WC1 (bitwise) QP 1152-1183 1183. Bit [i] provides the interrupt status for RQ(i+1152) where i=0
(RQINTSTS37) to 31.[i]: Interrupt status for RQ(i+1152)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1184 to
0x10_0224 WC1 (bitwise) QP 1184-1215 1215. Bit [i] provides the interrupt status for RQ(i+1184) where i=0
(RQINTSTS38) to 31.[i]: Interrupt status for RQ(i+1184)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1216 to
0x10_0228 WC1 (bitwise) QP 1216-1247 1247. Bit [i] provides the interrupt status for RQ(i+1216) where i=0
(RQINTSTS39) to 31.[i]: Interrupt status for RQ(i+1216)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1248 to
0x10_022C WC1 (bitwise) QP 1248-1279 1279. Bit [i] provides the interrupt status for RQ(i+1248) where i=0
(RQINTSTS40) to 31.[i]: Interrupt status for RQ(i+1248)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1280 to
0x10_0230 WC1 (bitwise) QP 1280-1311 1311. Bit [i] provides the interrupt status for RQ(i+1280) where i=0
(RQINTSTS41) to 31.[i]: Interrupt status for RQ(i+1280)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1312 to
0x10_0234 WC1 (bitwise) QP 1312-1343 1343. Bit [i] provides the interrupt status for RQ(i+1312) where i=0
(RQINTSTS42) to 31.[i]: Interrupt status for RQ(i+1312)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1344 to
0x10_0238 WC1 (bitwise) QP 1344-1375 1375. Bit [i] provides the interrupt status for RQ(i+1344) where i=0
(RQINTSTS43) to 31.[i]: Interrupt status for RQ(i+1344)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1376 to
0x10_023C WC1 (bitwise) QP 1376-1407 1407. Bit [i] provides the interrupt status for RQ(i+1376) where i=0
(RQINTSTS44) to 31.[i]: Interrupt status for RQ(i+1376)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1408 to
0x10_0240 WC1 (bitwise) QP 1408-1439 1439. Bit [i] provides the interrupt status for RQ(i+1408) where i=0
(RQINTSTS45) to 31.[i]: Interrupt status for RQ(i+1408)

ERNIC v4.0 40
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1440 to
0x10_0244 WC1 (bitwise) QP 1440-1471 1471. Bit [i] provides the interrupt status for RQ(i+1440) where i=0
(RQINTSTS46) to 31.[i]: Interrupt status for RQ(i+1440)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1472 to
0x10_0248 WC1 (bitwise) QP 1472-1503 1503. Bit [i] provides the interrupt status for RQ(i+1472) where i=0
(RQINTSTS47) to 31.[i]: Interrupt status for RQ(i+1472)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1504 to
0x10_024C WC1 (bitwise) QP 1504-1535 1535. Bit [i] provides the interrupt status for RQ(i+1504) where i=0
(RQINTSTS48) to 31.[i]: Interrupt status for RQ(i+1504)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1536 to
0x10_0250 WC1 (bitwise) QP 1536-1567 1567. Bit [i] provides the interrupt status for RQ(i+1536) where i=0
(RQINTSTS49) to 31.[i]: Interrupt status for RQ(i+1536)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1568 to
0x10_0254 WC1 (bitwise) QP 1568-1599 1599. Bit [i] provides the interrupt status for RQ(i+1568) where i=0
(RQINTSTS50) to 31.[i]: Interrupt status for RQ(i+1568)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1600 to
0x10_0258 WC1 (bitwise) QP 1600-1631 1632. Bit [i] provides the interrupt status for RQ(i+1600) where i=0
(RQINTSTS51) to 31.[i]: Interrupt status for RQ(i+1600)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1632 to
0x10_025C WC1 (bitwise) QP 1632-1663 1663. Bit [i] provides the interrupt status for RQ(i+1632) where i=0
(RQINTSTS52) to 31.[i]: Interrupt status for RQ(i+1632)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1664 to
0x10_0260 WC1 (bitwise) QP 1664-1695 1695. Bit [i] provides the interrupt status for RQ(i+1664) where i=0
(RQINTSTS53) to 31.[i]: Interrupt status for RQ(i+1664)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1696 to
0x10_0264 WC1 (bitwise) QP 1696-1727 1727. Bit [i] provides the interrupt status for RQ(i+1696) where i=0
(RQINTSTS54) to 31.[i]: Interrupt status for RQ(i+1696)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1728 to
0x10_0268 WC1 (bitwise) QP 1728-1759 1759. Bit [i] provides the interrupt status for RQ(i+1728) where i=0
(RQINTSTS55) to 31.[i]: Interrupt status for RQ(i+1728)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1760 to
0x10_026C WC1 (bitwise) QP 1760-1791 1791. Bit [i] provides the interrupt status for RQ(i+1760) where i=0
(RQINTSTS56) to 31.[i]: Interrupt status for RQ(i+1760)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1792 to
0x10_0270 WC1 (bitwise) QP 1792-1823 1823. Bit [i] provides the interrupt status for RQ(i+1792) where i=0
(RQINTSTS57) to 31.[i]: Interrupt status for RQ(i+1792)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1824 to
0x10_0274 WC1 (bitwise) QP 1824-1855 1855. Bit [i] provides the interrupt status for RQ(i+1824) where i=0
(RQINTSTS58) to 31.[i]: Interrupt status for RQ(i+1824)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1856 to
0x10_0278 WC1 (bitwise) QP 1856-1887 1887. Bit [i] provides the interrupt status for RQ(i+1856) where i=0
(RQINTSTS59) to 31.[i]: Interrupt status for RQ(i+1856)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1888 to
0x10_027C WC1 (bitwise) QP 1888-1919 1919. Bit [i] provides the interrupt status for RQ(i+1888) where i=0
(RQINTSTS60) to 31.[i]: Interrupt status for RQ(i+1888)

ERNIC v4.0 41
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1920 to
0x10_0280 WC1 (bitwise) QP 1920-1951 1951. Bit [i] provides the interrupt status for RQ(i+1920) where i=0
(RQINTSTS61) to 31.[i]: Interrupt status for RQ(i+1920)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1952 to
0x10_0284 WC1 (bitwise) QP 1952-1983 1983. Bit [i] provides the interrupt status for RQ(i+1952) where i=0
(RQINTSTS62) to 31.[i]: Interrupt status for RQ(i+1952)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 1984 to
0x10_0288 WC1 (bitwise) QP 1984-2015 2016. Bit [i] provides the interrupt status for RQ(i+1984) where i=0
(RQINTSTS63) to 31.[i]: Interrupt status for RQ(i+1984)
RCV Q interrupt status This register provides the RQ interrupt status for QPs 2016 to
0x10_028C WC1 (bitwise) QP 2016-2047 2047. Bit [i] provides the interrupt status for RQ(i+2016) where i=0
(RQINTSTS64) to 31.[i]: Interrupt status for RQ(i+2016)
Completion Q interrupt This register provides the CQ interrupt status for QPs 0 to 31. Bit
0x10_0290 WC1 status (bitwise) QP 0-31 [i] provides the interrupt status for CQi where i=0 to 31.[i]:
(CQINTSTS1) Interrupt status for CQ i
Completion Q interrupt This register provides the CQ interrupt status for QPs 32 to 63. Bit
0x10_0294 WC1 status (bitwise) QP 32-63 [i] provides the interrupt status for CQ(i+32) where i=0 to 31.[i]:
(CQINTSTS2) Interrupt status for CQ(i+32)
Completion Q interrupt This register provides the CQ interrupt status for QPs 64 to 95. Bit
0x10_0298 WC1 status (bitwise) QP 64-95 [i] provides the interrupt status for CQ(i+64) where i=0 to 31.[i]:
(CQINTSTS3) Interrupt status for CQ(i+64)
Completion Q interrupt This register provides the CQ interrupt status for QPs 96 to 127. Bit
0x10_029C WC1 status (bitwise) QP 96-127 [i] provides the interrupt status for CQ(i+92) where i=0 to 31.[i]:
(CQINTSTS4) Interrupt status for CQ(i+92)
Completion Q interrupt This register provides the CQ interrupt status for QPs 128 to 159.
0x10_02A0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+128) where i=0 to
128-159 (CQINTSTS5) 31.[i]: Interrupt status for CQ(i+128)
Completion Q interrupt This register provides the CQ interrupt status for QPs 160 to 191.
0x10_02A4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+160) where i=0 to
160-191 (CQINTSTS6) 31.[i]: Interrupt status for CQ(i+160)
Completion Q interrupt This register provides the CQ interrupt status for QPs 192 to 223.
0x10_02A8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+192) where i=0 to
192-223 (CQINTSTS7) 31.[i]: Interrupt status for CQ(i+192)
Completion Q interrupt This register provides the CQ interrupt status for QPs 224 to 255.
0x10_02AC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+224) where i=0 to
224-255 (CQINTSTS8) 31.[i]: Interrupt status for CQ(i+224)
Completion Q interrupt This register provides the CQ interrupt status for QPs 256 to 287.
0x10_02B0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+256) where i=0 to
256-287 (CQINTSTS9) 31.[i]: Interrupt status for CQ(i+256)
Completion Q interrupt This register provides the CQ interrupt status for QPs 288 to 319.
0x10_02B4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+288) where i=0 to
288-319 (CQINTSTS10) 31.[i]: Interrupt status for CQ(i+288)
Completion Q interrupt This register provides the CQ interrupt status for QPs 320 to 351.
0x10_02B8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+320) where i=0 to
320-351 (CQINTSTS11) 31.[i]: Interrupt status for CQ(i+320)

ERNIC v4.0 42
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
Completion Q interrupt This register provides the CQ interrupt status for QPs 352 to 383.
0x10_02BC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+352) where i=0 to
352-383 (CQINTSTS12) 31.[i]: Interrupt status for CQ(i+352)
Completion Q interrupt This register provides the CQ interrupt status for QPs 384 to 415.
0x10_02C0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+384) where i=0 to
384-415 (CQINTSTS13) 31.[i]: Interrupt status for CQ(i+384)
Completion Q interrupt This register provides the CQ interrupt status for QPs 416 to 447.
0x10_02C4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+416) where i=0 to
416-447 (CQINTSTS14) 31.[i]: Interrupt status for CQ(i+416)
Completion Q interrupt This register provides the CQ interrupt status for QPs 448 to 479.
0x10_02C8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+448) where i=0 to
448-479 (CQINTSTS15) 31.[i]: Interrupt status for CQ(i+448)
Completion Q interrupt This register provides the CQ interrupt status for QPs 480 to 511.
0x10_02CC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+480) where i=0 to
480-511 (CQINTSTS16) 31.[i]: Interrupt status for CQ(i+480)
Completion Q interrupt This register provides the CQ interrupt status for QPs 512 to 543.
0x10_02D0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+512) where i=0 to
512-543 (CQINTSTS17) 31.[i]: Interrupt status for CQ(i+512)
Completion Q interrupt This register provides the CQ interrupt status for QPs 544 to 575.
0x10_02D4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+544) where i=0 to
544-575 (CQINTSTS18) 31.[i]: Interrupt status for CQ(i+544)
Completion Q interrupt This register provides the CQ interrupt status for QPs 576 to 607.
0x10_02D8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+576) where i=0 to
576-607 (CQINTSTS19) 31.[i]: Interrupt status for CQ(i+576)
Completion Q interrupt This register provides the CQ interrupt status for QPs 608 to 639.
0x10_02DC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+608) where i=0 to
608-639 (CQINTSTS20) 31.[i]: Interrupt status for CQ(i+608)
Completion Q interrupt This register provides the CQ interrupt status for QPs 640 to 671.
0x10_02E0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+640) where i=0 to
640-671 (CQINTSTS21) 31.[i]: Interrupt status for CQ(i+640)
Completion Q interrupt This register provides the CQ interrupt status for QPs 672 to 703.
0x10_02E4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+672) where i=0 to
672-703 (CQINTSTS22) 31.[i]: Interrupt status for CQ(i+672)
Completion Q interrupt This register provides the CQ interrupt status for QPs 704 to 735.
0x10_02E8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+704) where i=0 to
704-735 (CQINTSTS23) 31.[i]: Interrupt status for CQ(i+704)
Completion Q interrupt This register provides the CQ interrupt status for QPs 736 to 767.
0x10_02EC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+736) where i=0 to
736-767 (CQINTSTS24) 31.[i]: Interrupt status for CQ(i+736)
Completion Q interrupt This register provides the CQ interrupt status for QPs 768 to 799.
0x10_02F0 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+768) where i=0 to
768-799 (CQINTSTS25) 31.[i]: Interrupt status for CQ(i+768)
Completion Q interrupt This register provides the CQ interrupt status for QPs 800 to 831.
0x10_02F4 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+800) where i=0 to
800-831 (CQINTSTS26) 31.[i]: Interrupt status for CQ(i+800)

ERNIC v4.0 43
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
Completion Q interrupt This register provides the CQ interrupt status for QPs 832 to 863.
0x10_02F8 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+832) where i=0 to
832-863 (CQINTSTS27) 31.[i]: Interrupt status for CQ(i+832)
Completion Q interrupt This register provides the CQ interrupt status for QPs 864 to 895.
0x10_02FC WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+864) where i=0 to
864-895 (CQINTSTS28) 31.[i]: Interrupt status for CQ(i+864)
Completion Q interrupt This register provides the CQ interrupt status for QPs 896 to 927.
0x10_0300 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+896) where i=0 to
896-927 (CQINTSTS29) 31.[i]: Interrupt status for CQ(i+896)
Completion Q interrupt This register provides the CQ interrupt status for QPs 928 to 949.
0x10_0304 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+928) where i=0 to
928-949 (CQINTSTS30) 31.[i]: Interrupt status for CQ(i+928)
Completion Q interrupt This register provides the CQ interrupt status for QPs 960 to 991.
0x10_0308 WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+960) where i=0 to
960-991 (CQINTSTS31) 31.[i]: Interrupt status for CQ(i+960)
Completion Q interrupt This register provides the CQ interrupt status for QPs 992 to 1023.
0x10_030C WC1 status (bitwise) QP Bit [i] provides the interrupt status for CQ(i+992) where i=0 to
992-1023 (CQINTSTS32) 31.[i]: Interrupt status for CQ(i+992)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1024 to
0x10_0310 WC1 status (bitwise) QP 1055. Bit [i] provides the interrupt status for CQ(i+1024) where i=0
1024-1055 (CQINTSTS33) to 31.[i]: Interrupt status for CQ(i+1024)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1056 to
0x10_0314 WC1 status (bitwise) QP 1087. Bit [i] provides the interrupt status for CQ(i+1056) where i=0
1056-1087 (CQINTSTS34) to 31.[i]: Interrupt status for CQ(i+1056)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1088 to
0x10_0318 WC1 status (bitwise) QP 1119. Bit [i] provides the interrupt status for CQ(i+1088) where i=0
1088-1119 (CQINTSTS35) to 31.[i]: Interrupt status for CQ(i+1088)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1120 to
0x10_031C WC1 status (bitwise) QP 1151. Bit [i] provides the interrupt status for CQ(i+1120) where i=0
1120-1151 (CQINTSTS36) to 31.[i]: Interrupt status for CQ(i+1120)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1152 to
0x10_0320 WC1 status (bitwise) QP 1183. Bit [i] provides the interrupt status for CQ(i+1152) where i=0
1152-1183 (CQINTSTS37) to 31.[i]: Interrupt status for CQ(i+1152)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1184 to
0x10_0324 WC1 status (bitwise) QP 1215. Bit [i] provides the interrupt status for CQ(i+1184) where i=0
1184-1215 (CQINTSTS38) to 31.[i]: Interrupt status for CQ(i+1184)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1216 to
0x10_0328 WC1 status (bitwise) QP 1247. Bit [i] provides the interrupt status for CQ(i+1216) where i=0
1216-1247 (CQINTSTS39) to 31.[i]: Interrupt status for CQ(i+1216)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1248 to
0x10_032C WC1 status (bitwise) QP 1279. Bit [i] provides the interrupt status for CQ(i+1248) where i=0
1248-1279 (CQINTSTS40) to 31.[i]: Interrupt status for CQ(i+1248)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1280 to
0x10_0330 WC1 status (bitwise) QP 1311. Bit [i] provides the interrupt status for CQ(i+1280) where i=0
1280-1311 (CQINTSTS41) to 31.[i]: Interrupt status for CQ(i+1280)

ERNIC v4.0 44
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
Completion Q interrupt This register provides the CQ interrupt status for QPs 1312 to
0x10_0334 WC1 status (bitwise) QP 1343. Bit [i] provides the interrupt status for CQ(i+1312) where i=0
1312-1343 (CQINTSTS42) to 31.[i]: Interrupt status for CQ(i+1312)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1344 to
0x10_0338 WC1 status (bitwise) QP 1375. Bit [i] provides the interrupt status for CQ(i+1344) where i=0
1344-1375 (CQINTSTS43) to 31.[i]: Interrupt status for CQ(i+1344)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1376 to
0x10_033C WC1 status (bitwise) QP 1407. Bit [i] provides the interrupt status for CQ(i+1376) where i=0
1376-1407 (CQINTSTS44) to 31.[i]: Interrupt status for CQ(i+1376)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1408 to
0x10_0340 WC1 status (bitwise) QP 1439. Bit [i] provides the interrupt status for CQ(i+1408) where i=0
1408-1439 (CQINTSTS45) to 31.[i]: Interrupt status for CQ(i+1408)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1440 to
0x10_0344 WC1 status (bitwise) QP 1471. Bit [i] provides the interrupt status for CQ(i+1440) where i=0
1440-1471 (CQINTSTS46) to 31.[i]: Interrupt status for CQ(i+1440)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1472 to
0x10_0348 WC1 status (bitwise) QP 1503. Bit [i] provides the interrupt status for CQ(i+1472) where i=0
1472-1503 (CQINTSTS47) to 31.[i]: Interrupt status for CQ(i+1472)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1504 to
0x10_034C WC1 status (bitwise) QP 1535. Bit [i] provides the interrupt status for CQ(i+1504) where i=0
1504-1535 (CQINTSTS48) to 31.[i]: Interrupt status for CQ(i+1504)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1536 to
0x10_0350 WC1 status (bitwise) QP 1567. Bit [i] provides the interrupt status for CQ(i+1536) where i=0
1536-1567 (CQINTSTS49) to 31.[i]: Interrupt status for CQ(i+1536)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1568 to
0x10_0354 WC1 status (bitwise) QP 1599. Bit [i] provides the interrupt status for CQ(i+1568) where i=0
1568-1599 (CQINTSTS50) to 31.[i]: Interrupt status for CQ(i+1568)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1600 to
0x10_0358 WC1 status (bitwise) QP 1632. Bit [i] provides the interrupt status for CQ(i+1600) where i=0
1600-1631 (CQINTSTS51) to 31.[i]: Interrupt status for CQ(i+1600)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1632 to
0x10_035C WC1 status (bitwise) QP 1663. Bit [i] provides the interrupt status for CQ(i+1632) where i=0
1632-1663 (CQINTSTS52) to 31.[i]: Interrupt status for CQ(i+1632)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1664 to
0x10_0360 WC1 status (bitwise) QP 1695. Bit [i] provides the interrupt status for CQ(i+1664) where i=0
1664-1695 (CQINTSTS53) to 31.[i]: Interrupt status for CQ(i+1664)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1696 to
0x10_0364 WC1 status (bitwise) QP 1727. Bit [i] provides the interrupt status for CQ(i+1696) where i=0
1696-1727 (CQINTSTS54) to 31.[i]: Interrupt status for CQ(i+1696)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1728 to
0x10_0368 WC1 status (bitwise) QP 1759. Bit [i] provides the interrupt status for CQ(i+1728) where i=0
1728-1759 (CQINTSTS55) to 31.[i]: Interrupt status for CQ(i+1728)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1760 to
0x10_036C WC1 status (bitwise) QP 1791. Bit [i] provides the interrupt status for CQ(i+1760) where i=0
1760-1791 (CQINTSTS56) to 31.[i]: Interrupt status for CQ(i+1760)

ERNIC v4.0 45
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
Completion Q interrupt This register provides the CQ interrupt status for QPs 1792 to
0x10_0370 WC1 status (bitwise) QP 1823. Bit [i] provides the interrupt status for CQ(i+1792) where i=0
1792-1823 (CQINTSTS57) to 31.[i]: Interrupt status for CQ(i+1792)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1824 to
0x10_0374 WC1 status (bitwise) QP 1855. Bit [i] provides the interrupt status for CQ(i+1824) where i=0
1824-1855 (CQINTSTS58) to 31.[i]: Interrupt status for CQ(i+1824)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1856 to
0x10_0378 WC1 status (bitwise) QP 1887. Bit [i] provides the interrupt status for CQ(i+1856) where i=0
1856-1887 (CQINTSTS59) to 31.[i]: Interrupt status for CQ(i+1856)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1888 to
0x10_037C WC1 status (bitwise) QP 1919. Bit [i] provides the interrupt status for CQ(i+1888) where i=0
1888-1919 (CQINTSTS60) to 31.[i]: Interrupt status for CQ(i+1888)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1920 to
0x10_0380 WC1 status (bitwise) QP 1951. Bit [i] provides the interrupt status for CQ(i+1920) where i=0
1920-1951 (CQINTSTS61) to 31.[i]: Interrupt status for CQ(i+1920)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1952 to
0x10_0384 WC1 status (bitwise) QP 1983. Bit [i] provides the interrupt status for CQ(i+1952) where i=0
1952-1983 (CQINTSTS62) to 31.[i]: Interrupt status for CQ(i+1952)
Completion Q interrupt This register provides the CQ interrupt status for QPs 1984 to
0x10_0388 WC1 status (bitwise) QP 2016. Bit [i] provides the interrupt status for CQ(i+1984) where i=0
1984-2015 (CQINTSTS63) to 31.[i]: Interrupt status for CQ(i+1984)
Completion Q interrupt This register provides the CQ interrupt status for QPs 2016 to
0x10_038C WC1 status (bitwise) QP 2047. Bit [i] provides the interrupt status for CQ(i+2016) where i=0
2016-2047 (CQINTSTS64) to 31.[i]: Interrupt status for CQ(i+2016)
CNP Packed scheduled
interrupt status (bitwise)
0x10_0390 RO This register provides the CNP scheduled interrupt for QPs 0 to 31
QP 0-31
(CNPSCHDSTS1REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 32 to
0x10_0394 RO
QP 32-63 63
(CNPSCHDSTS2REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 64 to
0x10_0398 RO
QP 64-95 95
(CNPSCHDSTS3REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 96 to
0x10_039C RO
QP 96-127 127
(CNPSCHDSTS4REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 128 to
0x10_03A0 RO
QP 128-159 159
(CNPSCHDSTS5REG)

ERNIC v4.0 46
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 160 to
0x10_03A4 RO
QP 160-191 191
(CNPSCHDSTS6REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 192 to
0x10_03A8 RO
QP 192-223 223
(CNPSCHDSTS7REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 224 to
0x10_03AC RO
QP 224-255 255
(CNPSCHDSTS8REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 256 to
0x10_03B0 RO
QP 256-287 287
(CNPSCHDSTS9REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 288 to
0x10_03B4 RO
QP 288-319 319
(CNPSCHDSTS10REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 320 to
0x10_03B8 RO
QP 320-351 351
(CNPSCHDSTS11REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 352 to
0x10_03BC RO
QP 352-383 383
(CNPSCHDSTS12REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 384 to
0x10_03C0 RO
QP 384-415 415
(CNPSCHDSTS13REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 416 to
0x10_03C4 RO
QP 416-447 447
(CNPSCHDSTS14REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 448 to
0x10_03C8 RO
QP 448-479 479
(CNPSCHDSTS15REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 480 to
0x10_03CC RO
QP 480-511 511
(CNPSCHDSTS16REG)

ERNIC v4.0 47
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 512 to
0x10_03D0 RO
QP 512-543 543
(CNPSCHDSTS17REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 544 to
0x10_03D4 RO
QP 544-575 575
(CNPSCHDSTS18REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 576 to
0x10_03D8 RO
QP 576-607 607
(CNPSCHDSTS19REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 608 to
0x10_03DC RO
QP 608-639 639
(CNPSCHDSTS20REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 640 to
0x10_03E0 RO
QP 640-671 671
(CNPSCHDSTS21REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 672 to
0x10_03E4 RO
QP 672-703 703
(CNPSCHDSTS22REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 704 to
0x10_03E8 RO
QP 704-735 735
(CNPSCHDSTS23REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 736 to
0x10_03EC RO
QP 736-767 767
(CNPSCHDSTS24REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 768 to
0x10_03F0 RO
QP 768-799 799
(CNPSCHDSTS25REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 800 to
0x10_03F4 RO
QP 800-831 831
(CNPSCHDSTS26REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 832 to
0x10_03F8 RO
QP 832-863 863
(CNPSCHDSTS27REG)

ERNIC v4.0 48
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 864 to
0x10_03FC RO
QP 864-895 895
(CNPSCHDSTS28REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 896 to
0x10_0400 RO
QP 896-927 927
(CNPSCHDSTS29REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 928 to
0x10_0404 RO
QP 928-949 949
(CNPSCHDSTS30REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 960 to
0x10_0408 RO
QP 960-991 991
(CNPSCHDSTS31REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 992 to
0x10_040C RO
QP 992-1023 1023
(CNPSCHDSTS32REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1024
0x10_0410 RO
QP 1024-1055 to 1055
(CNPSCHDSTS33REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1056
0x10_0414 RO
QP 1056-1087 to 1087
(CNPSCHDSTS34REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1088
0x10_0418 RO
QP 1088-1119 to 1119
(CNPSCHDSTS35REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1120
0x10_041C RO
QP 1120-1151 to 1151
(CNPSCHDSTS36REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1152
0x10_0420 RO
QP 1152-1183 to 1183
(CNPSCHDSTS37REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1184
0x10_0424 RO
QP 1184-1215 to 1215
(CNPSCHDSTS38REG)

ERNIC v4.0 49
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1216
0x10_0428 RO
QP 1216-1247 to 1247
(CNPSCHDSTS39REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1248
0x10_042C RO
QP 1248-1279 to 1279
(CNPSCHDSTS40REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1280
0x10_0430 RO
QP 1280-1311 to 1311
(CNPSCHDSTS41REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1312
0x10_0434 RO
QP 1312-1343 to 1343
(CNPSCHDSTS42REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1344
0x10_0438 RO
QP 1344-1375 to 1375
(CNPSCHDSTS43REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1376
0x10_043C RO
QP 1376-1407 to 1407
(CNPSCHDSTS44REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1408
0x10_0440 RO
QP 1408-1439 to 1439
(CNPSCHDSTS45REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1440
0x10_0444 RO
QP 1440-1471 to 1471
(CNPSCHDSTS46REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1472
0x10_0448 RO
QP 1472-1503 to 1503
(CNPSCHDSTS47REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1504
0x10_044C RO
QP 1504-1535 to 1535
(CNPSCHDSTS48REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1536
0x10_0450 RO
QP 1536-1567 to 1567
(CNPSCHDSTS49REG)

ERNIC v4.0 50
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1568
0x10_0454 RO
QP 1568-1599 to 1599
(CNPSCHDSTS50REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1600
0x10_0458 RO
QP 1600-1631 to 1631
(CNPSCHDSTS51REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1632
0x10_045C RO
QP 1632-1663 to 1663
(CNPSCHDSTS52REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1664
0x10_0460 RO
QP 1664-1695 to 1695
(CNPSCHDSTS53REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1696
0x10_0464 RO
QP 1696-1727 to 1727
(CNPSCHDSTS54REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1728
0x10_0468 RO
QP 1728-1759 to 1759
(CNPSCHDSTS55REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1760
0x10_046C RO
QP 1760-1791 to 1791
(CNPSCHDSTS56REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1792
0x10_0470 RO
QP 1792-1823 to 1823
(CNPSCHDSTS57REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1824
0x10_0474 RO
QP 1824-1855 to 1855
(CNPSCHDSTS58REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1856
0x10_0478 RO
QP 1856-1887 to 1887
(CNPSCHDSTS59REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1888
0x10_047C RO
QP 1888-1919 to 1919
(CNPSCHDSTS60REG)

ERNIC v4.0 51
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1920
0x10_0480 RO
QP 1920-1951 to 1951
(CNPSCHDSTS61REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1952
0x10_0484 RO
QP 1952-1983 to 1983
(CNPSCHDSTS62REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 1984
0x10_0488 RO
QP 1984-2015 to 2015
(CNPSCHDSTS63REG)
CNP Packed scheduled
interrupt status (bitwise) This register provides the CNP scheduled interrupt for QPs 2016
0x10_048C RO
QP 2016-2047 to 2047
(CNPSCHDSTS64REG)

ERNIC v4.0 52
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
Per QP Registers
This register defines the basic configuration of QP. This register is
generated based on the parameter C_NUM_QP (where i = 1 to
C_NUM_QP). Register offset for per-QP is 0x18_0000, 0x18_0004
and so on for the subsequent offsets.
• [0]: QP enable – Should be set to 1 for all active QPs. A disabled
QP will not be able to receive or transmit packets.
• [2]: RQ interrupt enable – When enabled, allows the receive
queue interrupt to be generated for every new packet received
on the receive queue
• [3]: CQ interrupt enable – When enabled, allows the completion
queue interrupt to be generated for every send work queue
entry completion
• [4]: HW Handshake disable – This bit when reset to 0 enables the
HW handshake ports for doorbell exchange. If set, all doorbell
values are exchanged through writes through the AXI4 or
AXI4-Lite interface.
• [5]: CQE write enable – This bit when set, enables completion
0x18_0000 queue entry writes. The writes are disabled when this bit is reset.
QP Configuration QPi CQE writes can be enabled to debug failed completions.
+ ((i-1) x RW
(QPCONFi)
0x0100) • [6]: QP under recovery. This bit need to be set in the fatal
clearing process.
• [7]: QP configured for IPv4 or IPv6
° 0 - IPv4
° 1 - IPv6
• [10:8]: Path MTU
° 000 – 256B (default)
° 001 – 512B
° 010 – 1024B
° 011 – 2048B
° 100 - 4096B
° 101 to 111 - Reserved
• [31:16]: RQ Buffer size (in multiple of 256B). This is the size of
each buffer element in the request and not the size of the entire
request.
The programmed value should be the power of 2 for expected
behavior.
This register is generated based on the parameter C_NUM_QP
0x18_0004 QP advanced (where i = 2 to C_NUM_QP). This register is not present for QP1.
+ ((i-1) x RW configuration QPi • [5:0]: Traffic class (keep default reset value)
0x0100) (QPADVCONFi) • [15:8]: Time to live
• [31:16]: Partition Key
This register is generated based on the parameter C_NUM_QP
0x18_0008 (where i = 1 to C_NUM_QP). This register provides the base
RCV Q Buffer base
+ ((i-1) x RW address of the RDMA Receive queue buffer.
address QPi (RQBAi)
0x0100) [31:8]: Receive Q Buffer Base address addr (256 B aligned). 256 B
is the total (individual rq buffer element size)* rq_depth

ERNIC v4.0 53
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register is generated based on the parameter C_NUM_QP
0x18_00C0 (where i = 1 to C_NUM_QP). This register provides the msb base
RCV Q Buffer base
+ (i-1) x RW address of the RDMA Receive queue buffer.
address QPi (RQBAMSBi)
0x0100) [63:32]: Receive Q Buffer Base address addr (256 B aligned). 256 B
is the total (individual rq buffer element size)* rq_depth
This register is generated based on the parameter C_NUM_QP
0x18_0010
SEND Q base address QPi (where i = 1 to C_NUM_QP). This register provides the base
+ ((i-1) x RW
(SQBAi) address of the RDMA Send queue buffer.
0x0100)
[31:5]: Send Q base address (32 B aligned)
This register is generated based on the parameter C_NUM_QP
0x18_00C8
SEND Q base address msb (where i = 1 to C_NUM_QP). This register provides the msb base
+ ((i-1) x
QPi (SQBAMSBi) address of the RDMA Send queue buffer.
0x0100)
[63:32]: Send Q base address (32 B aligned)
This register is generated based on the parameter C_NUM_QP
0x18_0018
CQ base address QPi (where i = 1 to C_NUM_QP). This register provides the base
+ ((i-1) x RW
(CQBAi) address of the RDMA Completion queue buffer.
0x0100)
[31:5]: Send CQ base address (32 B aligned)
This register is generated based on the parameter C_NUM_QP
0x18_00D0
CQ base address msb QPi (where i = 1 to C_NUM_QP). This register provides the msb base
+ ((i-1) x RW
(CQBAMSBi) address of the RDMA Completion queue buffer.
0x0100)
[63:32]: Send CQ base address (32 B aligned)
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register provides the address of
0x18_0020 RCV Q Write pointer DB
the Receive Queue doorbell register. Upon reception of a new
+ ((i-1) x RW address QPi
incoming RDMA SEND packet, the ERNIC IP updates the RQ
0x0100) (RQWPTRDBADDi)
doorbell values in the address pointed to by this register.
[31:0]: RCV Q write pointer Doorbell address
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register provides the msb address
0x18_0024 RCV Q Write pointer DB
of the Receive Queue doorbell register. Upon reception of a new
+ ((i-1) x RW address msb QPi
incoming RDMA SEND packet, the ERNIC IP updates the RQ
0x0100) (RQWPTRDBADDMSBi)
doorbell values in the address pointed to by this register.
[63:32]: RCV Q write pointer Doorbell address msb
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). this register provides the address of
0x18_0028
CQ DB address QPi the Completion Queue doorbell register. Upon completion of a
+ ((i-1) x RW new SEND Work queue entry, the ERNIC IP updates the CQ
(CQDBADDi)
0x0100) doorbell values in the address pointed to by this register.
[31:0]: Send CQ Doorbell address
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). this register provides the msb address
0x18_002C
CQ DB address QPi of the Completion Queue doorbell register. Upon completion of a
+ ((i-1) x RW
(CQDBADDMSBi) new SEND Work queue entry, the ERNIC IP updates the CQ
0x0100)
doorbell values in the address pointed to by this register.
[63:32]: Send CQ Doorbell address

ERNIC v4.0 54
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This status register gives the Send
completion Queue doorbell value and provides information about
0x18_0030 the Send WQEs that have been completed. This is the doorbell
CQ head pointer QPi
+ ((i-1) x RO value that is written by the ERNIC IP to the address pointed to by
(CQHEADi)
0x0100) the CQDBADDi register.
• [15:0]: CQ head pointer
• [31:0]: Reserved
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register is updated by the target
0x18_0034 application on consuming a new receive queue entry. Once
RQ Consumer Index QPi consumed, the RQ entry can be over written by the ERNIC IP. The
+ ((i-1) x RW
(RQCIi) RQCI index can also be updated through the side band interface.
0x0100)
• [15:0]: RCV Q Consumer Index Doorbell
• [31:16]: Reserved
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register is updated by the target
application when it posts a new Send Work Queue entry. The SQPI
can also be updated through the side band interface. The Send
0x18_0038 Queue is a circular buffer. The CQHEAD register for the
SQ Producer index QPi
+ ((i-1) x RW corresponding QP provides information about the Work Queue
(SQPIi)
0x0100) entries that have been completed and can be over written by the
target application.
• [15:0]: Send Q Producer Index Doorbell
• [31:16]: Reserved
This register is generated based on the parameter C_NUM_QP
0x18_003C (where i = 1 to C_NUM_QP). This register defines the queue
+ ((i-1) x RW Q Depth QPi (QDEPTHi) depths for send, completion and receive queues.
0x0100) • [15:0]: Send Q depth (Send CQ will of the same depth)
• [31:16]: Receive Q depth
This register is initialized at connection time by the SW. After that
the HW updates it for every outgoing packet and should not be
0x18_0040
SEND Q PSN for QPi updated by the SW. This register is generated based on the
+ ((i-1) x RW
(SQPSNi) parameter C_NUM_QP (where i = 2 to C_NUM_QP). This register
0x0100)
does not exist for QP1.
[23:0]: Send Q PSN
This register provides the last incoming RQ packet details. This
0x18_0044 register is generated based on the parameter C_NUM_QP (where i
Last RQ req for QPi = 2 to C_NUM_QP). This register does not exist for QP1.
+ ((i-1) x RW
(LSTRQREQi)
0x0100) • [23:0]: RCV Q PSN
• [31:24]: RCV Q opcode
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register is configured at
0x18_0048 Destination QP
connection time by the SW and provides the remote QPID
+ ((i-1) x RW configuration for QPi
connected to this QP. All outgoing packets from this QP are sent
0x0100) (DESTQPCONFi) with this QPID as the destination QPID.
[23:0]: Destination Connected QPID

ERNIC v4.0 55
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register is configured at
0x18_0050 MAC destination address connection time by the SW and provides the MAC address of the
+ ((i-1) x RW LSB QPi remote host connected to this QP. All outgoing packets from this
0x0100) (MACDESADDLSBi) QP are sent with this MAC address (LSB) as the destination MAC
address.
[31:0]: MAC destination address LSB
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). This register is configured at
0x18_0054 MAC destination address connection time by the SW and provides the MAC address of the
+ ((i-1) x RW MSB QPi remote host connected to this QP. All outgoing packets from this
0x0100) (MACDESADDMSBi) QP are sent with this MAC address (MSB) as the destination MAC
address.
[15:0]: MAC destination address MSB
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). If the remote host for this QP is
0x18_0060
IP destination address 1 configured using the IPv4 protocol, then this register indicates an
+ ((i-1) x RW
QPi (IPDESADDR1i) IPv4 address. If the remote host is configured using the IPv6
0x0100)
protocol, this register then indicates the IPv6 destination address.
[31:0]: IP Destination address LSB1
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). If the remote host is configured using
0x18_0064
IP destination address 2 the IPv6 protocol, this register indicates an IPv6 destination
+ ((i-1) x RW
QPi (IPDESADDR2i) address [63:32]. If the remote host of this QP is configured using
0x0100)
the IPv4 protocol, then this register is not used.
[31:0]: IP Destination address LSB2
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). If the remote host is configured using
0x18_0068
IP destination address 3 the IPv6 protocol, this register indicates an IPv6 destination
+ ((i-1) x RW
QPi (IPDESADDR3i) address [95:64]. IF the remote host of this QP is configured using
0x0100)
the IPv4 protocol, then this register is not used.
[31:0]: IP Destination address MSB1
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). If the remote host is configured using
0x18_006C
IP destination address 4 the IPv6 protocol, this register indicates an IPv6 destination
+ ((i-1) x RW
QPi (IPDESADDR4i) address [127:96]. If the remote host of this QP is configured using
0x0100)
the IPv4 protocol, then this register is not used.
[31:0]: IP Destination address MSB2

ERNIC v4.0 56
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
This register is generated based on the parameter C_NUM_QP
(where i = 1 to C_NUM_QP). It provides the timeout configuration
QPi. This register does not exist for QP1.
• [5:0]: Timeout value
0x18_004C Timeout Configuration • [7:6]: Reserved
+ ((i-1) x RW Register for QPi • [10:8]: Maximum retry count
0x0100) (TIMEOUTCONFi)
• [13:11]: Maximum RNR retry count
• [15:14]: Reserved
• [20:16]: RNR NACK Timeout value for outgoing packets
• [31:21]: Reserved
0x18_0080 This register is generated based on the parameter C_NUM_QP
Status Sender sequence
+ ((i-1) x RO (where i = 2 to C_NUM_QP). This register does not exist for QP1.
number QPi (STATSSNi)
0x0100) [23:0]: Current outgoing SSN (same as outgoing MSN)
0x18_0084 This register is generated based on the parameter C_NUM_QP
Status Message sequence
+ ((i-1) x RO (where i = 2 to C_NUM_QP). This register does not exist for QP1.
number QPi (STATMSN)
0x0100) [23:0]: Current expected incoming MSN
This register provides the QP status for every QP. This register is
generated based on the parameter C_NUM_QP (where i = 1 to
C_NUM_QP).
• [0]: QP in fatal status
• [1]: RCV Q ovfl (outgoing RNR NACK)
• [2]: Send Q full
• [3]: Outstanding Q full
• [4]: CQ FIFO full
• [8:5]: Reserved
• [9]: Send Q empty. This bit signifies that there are no SQEs left
0x18_0088 to process. However, it does not imply that all the SEND WQEs
+ ((i-1) x RO Status QPi (STATQPi) were acknowledged by the remote host.
0x0100)
• [10]: Outstanding Q empty
• [11]: QP packet was retried
• [15:12]: Reserved
• [22:16]: NACK syndrome received (Read/Write)
• [23]: Reserved
• [26:24]: Current retry count (Read/Write)
• [27]: Reserved
• [30:28]: Current RNR nack count for inc resp packets (Read/
Write)
• [31]: Reserved
This register is generated based on the parameter C_NUM_QP
0x18_008C Status Current SQ ptr (where i = 2 to C_NUM_QP).
+ ((i-1) x RO under process [15:0]: Current SQ pointer that is under process. This shows the
0x0100) (STATCURSQPTRi) number of WQEs that are outstanding and awaiting a response
from the remote host.

ERNIC v4.0 57
PG332 December 2, 2022 www.xilinx.com
 Chapter 2: Product Specification

Table 2‐7: ERNIC Registers Details (Cont’d)

Address Access Register Name Details
0x18_0090 This register is generated based on the parameter C_NUM_QP
Status Response PSN for
+ ((i-1) x RO (where i = 2 to C_NUM_QP). This register does not exist for QP1.
QPi (STATRESPSNi)
0x0100) [23:0]: Expected response PSN
0x18_0094 Status RQ buffer current This register is generated based on the parameter C_NUM_QP
+ ((i-1) x RO address for QPi (where i = 2 to C_NUM_QP).
0x0100) (STATRQBUFCAi) [31:8]: Receive Q Buffer current addr (256 B aligned)
0x18_00D8 Status RQ buffer current This register is generated based on the parameter C_NUM_QP
+ ((i-1) x RO msb address for QPi (where i = 2 to C_NUM_QP).
0x0100) (STATRQBUFCAMSBi) [63:32]: Receive Q Buffer current addr msb (256 B aligned)
0x18_0098 This register is generated based on the parameter C_NUM_QP
Status of WQEs posted to
+ ((i-1) x RO (where i = 2 to C_NUM_QP).
QPi (STATWQEi)
0x0100) [15:0]: Count of WQEs pushed by QP MGR for this QP
0x18_009C This register is generated based on the parameter C_NUM_QP
Status RQ Producer index
+ ((i-1) x RO (where i = 2 to C_NUM_QP).
DB QPi (STATRQPIDBi)
0x0100) [15:0]: RCV Q Producer Index DB
This register is generated based on the parameter C_NUM_QP
0x18_00B0
Protection domain (where i = 2 to C_NUM_QP). This register is 24-bit and contains the
+ ((i-1) x RW
number PD number assigned to the QP.
0x0100)
Note: Only one memory region can be associated per QP.

ERNIC v4.0 58
PG332 December 2, 2022 www.xilinx.com
 Chapter 3

Designing with the Core

This chapter includes guidelines and additional information to facilitate designing with the
ERNIC IP core.

General Design Guidelines

A typical ERNIC subsystem would include one or more MAC hard/soft IPs. An AXI
interconnect would also be required to connect the various AXI interfaces exposed by the
ERNIC IP. A DRAM can be a part of the solution which would require a DRAM controller to
be instantiated as well.
X-Ref Target - Figure 3-1

DDR
wqe_proc_wr_retry
resp hndler l/f
pkt_hndler I/f

wQe_proc l/f

AXI4-Stream
QP mgr l/f

AXI4
AXI4-Lite
Customer side band
non_roce_dma_m_axis_I/f
M M M M M pkt_hndler rresp_I/f
AXI M
M
DMA non_roce_dma_s_axis_I/f
RQPI sideband
S M
RQCI sideband
ERNIC Target IP S Customer Logic
non_roce_cmac_s_axis_I/f
S SQCI sideband
roce_cmac_s_axis_I/f
M
CMAC S SQPI sideband
S
cmac_m_axis_I/f
M
S
s_axi_lite

X19883-010621

Figure 3‐1: ERNIC Interfaces

Figure 3-1 shows the various ERNIC IP interfaces. The interfaces shown as interfacing with
DDR may interface with any memory mapped region. Refer to Table 2-5 for details on each
of these interfaces. The AXI4 and the AXI4-Stream interfaces are 512 bits wide and are
mainly used for data transfers. The ERNIC IP provides sideband interfaces to allow for

ERNIC v4.0 59
PG332 December 2, 2022 www.xilinx.com
 Chapter 3: Designing with the Core

efficient exchange of queue pair related doorbells. These side band interfaces can be
enabled or disabled for each queue pair (QP) based on the configuration.

The ERNIC IP has one AXI4-Lite slave interface to access the register space. The details of
the memory map required for this slave interface is shown in the following table.

Table 3‐1: Address Space Allocation Requirement for Slave Interfaces

Slave Interface Size Unit Description
ERNIC AXI4-Lite slave interface 2 MB ERNIC register interface

Apart from these, the IP also requires some memory regions to be allocated for some
specific data structures. These memory regions may be mapped to a local DRAM or an AXI
BRAM or any other memory mapped slave. Ensure that there is adequate bandwidth on
these memory interfaces based on the line rate that ERNIC should achieve.

Table 3‐2: ERNIC IP Memory Requirement

Memory Region Size Unit Description
256 packets of 256 bytes each. Packets that fail packet
Error buffer 64 KB validation are sent to the error buffer along with 4
bytes of error syndrome
2048 QPs of depth 128 locations and each SQE is of
Send Queue 16 MB
64 bytes each
2048 QPs of depth 128 locations and each RQE is of
Receive Queue 256 MB
1024 bytes each
2048 QPs of depth 128 locations and each CQE is of 4
Send completion Queue 1 MB
bytes each
Buffers of 4K size each for 256 QPs and each QP with
Write Retry buffers 16 MB
up to 16 outstanding transactions

Notes:
1. Use the Vivado implementation strategy — Performance_RefinePlacement — when using the ERNIC with more
than 256 QPs.

Interrupts
ERNIC IP provides a single interrupt line, which is generated on different interrupts status
signals defined in INTEN register. Each of these interrupt lines can be enabled by writing a
1 to the corresponding bits of the Interrupt Enable INTEN register. On receiving an ORed
interrupt the SW can read the INTSTS register to know the cause of the interrupt.

Interrupt bits 4 and 6 inform the drivers about a WQE completion or an incoming RDMA
SEND respectively. These interrupts can be enabled or disabled on a per QP basis. Bit [2] of
the QPCONFi registers allows selective enabling of Receive Queue interrupts per QP.
Similarly, bit [3] of the QPCONFi registers allows selective enabling of Send Completion
Queue interrupts. In general QPs that require SW handling should have this option enabled.

ERNIC v4.0 60
PG332 December 2, 2022 www.xilinx.com
 Chapter 3: Designing with the Core

The QPs that are directly handled by the hardware will be informed through the hardware
handshake ports and corresponding interrupt enable bits can be disabled. Such QPs
should have the hardware handshake disable QPCONFi[4] bit reset to 0.

The RQINTSTSn and CQINTSTSn registers provide bitwise information about the QPs that
have a pending RQ or CQ entry to be serviced.

These registers should be read by the SW on receiving an RQ or CQ interrupt respectively to

know the QPs that are required to be serviced. These interrupt status should be cleared
upon successful handling by writing a 1 to the respective bits.

Clocking
Two clocks are exposed at the top of the ERNIC IP. These are: AXI4 clock and AXI4-Lite clock.
All the registers accesses work on the AXI4-Lite clock while the rest of the logic works on
AXI4 clock. Typically, the AXI4 clock would be of higher frequency (up to 200 MHz) while the
AXI4-Lite interface could be clocked at a lower frequency (divided AXI4 clock with the
divided clock edges aligning with the AXI4 clock). However, these clocks are treated as
synchronous inside the ERNIC IP and are expected to be generated from the same clock
source.

Resets
The ERNIC IP requires two active-Low resets that are synchronized to the two clock
domains, respectively.

ERNIC v4.0 61
PG332 December 2, 2022 www.xilinx.com
 Chapter 4

Design Flow Steps

This chapter describes customizing and generating the core, constraining the core, and the
simulation, synthesis and implementation steps that are specific to this IP core. More
detailed information about the standard Vivado® design flows and the IP integrator can be
found in the following Vivado Design Suite user guides:

• Vivado Design Suite User Guide: Designing IP Subsystems using IP Integrator (UG994)
[Ref 2]
• Vivado Design Suite User Guide: Designing with IP (UG896) [Ref 3]
• Vivado Design Suite User Guide: Getting Started (UG910) [Ref 4]
• Vivado Design Suite User Guide: Logic Simulation (UG900) [Ref 5]

Customizing and Generating the Core

This section includes information about using Xilinx tools to customize and generate the
core in the Vivado Design Suite.

If you are customizing and generating the core in the Vivado IP integrator, see the Vivado
Design Suite User Guide: Designing IP Subsystems using IP Integrator (UG994) [Ref 2] for
detailed information. IP integrator might auto-compute certain configuration values when
validating or generating the design. To check whether the values change, see the
description of the parameter in this chapter. To view the parameter value, run the
validate_bd_design command in the Tcl Console.

You can customize the IP for use in your design by specifying values for the various
parameters associated with the IP core using the following steps:

1. Select the IP from the Vivado IP catalog.

2. Double-click the selected IP or select the Customize IP command from the tool bar or
right-click menu.

For details, see the Vivado Design Suite User Guide: Designing with IP (UG896) [Ref 3] and
the Vivado Design Suite User Guide: Getting Started (UG910) [Ref 4].

Note: Figures in this chapter are illustrations of the Vivado Integrated Design Environment (IDE).
The layout depicted here might vary from the current version.

ERNIC v4.0 62
PG332 December 2, 2022 www.xilinx.com
 Chapter 4: Design Flow Steps

Output Generation
For details, see the Vivado Design Suite User Guide: Designing with IP (UG896) [Ref 3].

Simulation
For comprehensive information about Vivado simulation components, as well as
information about using supported third-party tools, see the Vivado Design Suite User
Guide: Logic Simulation (UG900) [Ref 5].

IMPORTANT: For cores targeting Xilinx 7 series FPGAs or Zynq-7000 devices, UNIFAST libraries are not
supported. Xilinx IP is tested and qualified with UNISIM libraries only.

Synthesis and Implementation

For details about synthesis and implementation, see the Vivado Design Suite User Guide:
Designing with IP (UG896) [Ref 3].

ERNIC v4.0 63
PG332 December 2, 2022 www.xilinx.com
 Chapter 5

Example Design
This chapter contains information about the ERNIC core example design. The example
design consists of the following modules:

• Clock and Reset Generator (simulation only)

• Register Configuration Module
• Send, Read Response, and ACK Generator
• WQE generator
• TX Path Checker
• RX Path Checker

Figure 5-1 shows the top-level example design architecture.

X-Ref Target - Figure 5-1

Memory Read
Legend from
[Header and Data] Memory
Example design modules

XRNIC IP Modules
AXI4-Stream I/F VerticalHeader
AXI4 I/F
AXI4-Lite I/F
Side Band I/F

DMA

Packet
Read Read SGL
Generator WQE Processor
[TX-WQE] WQE QP Manager
SQ-PI
BRAM
Update
Configuration TX Packet
Reg Config CRC Calculator Checker
Module Registers
Initiation
file

Write
Response Handler Completions Memory [Data]

Data
RX Checker Packet Generator
Data/Doorbell
towards RX Packet Handler [RX]
NVMof

Memory
[capsules towards
NVMof]

Figure 5‐1: Example Design Architecture

ERNIC v4.0 64
PG332 December 2, 2022 www.xilinx.com
 Chapter 5: Example Design

Apart from the ERNIC IP, the example design integrates the following modules:

• Register Configuration Module: This module configures all the required registers of
the ERNIC IP.
• Packet Generator Module: This generates the following types of packets:

° SEND Packets

° RDMA Read Response Packets for all the Read Requests from the ERNIC IP

° ACK Packets for all the RDMA Write requests from the ERNIC IP

° RDMA READ and RDMA WRITE

• Example Design Features: RDMA READ and RDMA WRITE
• RX Checker Module: This module checks the capsules (from SEND packets) and
payload (from Read Response) received from the ERNIC IP. This module also checks the
number of door bells rang on the RQ side band interface. The payload size of Read
Response packets, RDMA write requests is 256 bytes. Capsule size in the SEND packets
is 80 bytes.
• WQE Generator Module: This is responsible for generating the work queue requests
and SQ PI doorbell updates to the ERNIC IP.
• TX Checker Module: This checks the data transmitted by the ERNIC IP over the
streaming interface.
• Initiator WQE Module: This fetches the data from DDR when RDMA READ occurs.
• Initiator Checker Modules:

° Data checker which checks the data received for RDMA WRITE operation.

° RDMA READ RESPONSE packet check for RDMA READ operation and ACK packet
check for RDMA WRITE operation.

Example Design Features

The example design exercises the features of the ERNIC IP. The following incoming packets
are sent to the ERNIC IP:

• RDMA SEND
• RDMA Read Response
• ACK Packets
• RDMA READ
• RDMA WRITE

ERNIC v4.0 65
PG332 December 2, 2022 www.xilinx.com
 Chapter 5: Example Design

The following outgoing packets are checked by the example design:

• RDMA Read Requests

• RDMA Write Requests
• RDMA READ Response
• ACK Packets

The following are the hardware handshake paths of ERNIC:

• Only IPV4 Packets are exercised

• Only six QPs are exercised (QP2 to QP7)

Example Design Limitations

The ERNIC example design has the following limitations:

• Example design does not exercise Retry path

• Example design exercises only Hardware handshake path (QPCONFi[4] is set to 0 for all
QPs)
• Example design does not exercise IPV6 packets
• Example design does not generate any connection management (MAD) packets to QP1
• Example design limits the number of each supported packet type transactions to 8

Simulating the Example Design

For more information on Simulation, see the Vivado Design Suite User Guide: Logic
Simulation (UG900) [Ref 5].

Simulation Results
The simulation script compiles the ERNIC example design and supporting simulation files.
It then runs the simulation and checks to ensure that it completed successfully. If the test
passes, then the following message is displayed:

Test Completed Successfully

If the test fails, then the following message is displayed:

ERROR: Test Failed

ERNIC v4.0 66
PG332 December 2, 2022 www.xilinx.com
 Chapter 5: Example Design

If the test hangs, then the following message is displayed:

ERROR: Test did not complete (timed-out)

Example Sequence
The demonstration test bench performs the following tasks:

• All the required ERNIC registers are configured by the Register Configuration Module
through AXI4-Lite interface.
• There are three operations being handled in example test bench:
a. RDMA SEND:
- Packet Generator module generates eight SEND packets to the QP2 to QP7.
- RX Path Checker of example design, checks for the data integrity on the capsule
transferred from the ERNIC along with the door bells rang.
b. RDMA RD REQUEST:
- Example design has a predefined RDMA Read work Queue entry packets. The
WQE Generator module rings the SQPI doorbell and sends the work queue entry
packets when requested by the ERNIC.
- The TX Checker Module checks the necessary fields of the RDMA read request
received.
- After successful validation of the request, the Packet Generator module sends
RDMA Read Response to the ERNIC IP.
- The RX Path Checker checks for the data integrity on the payload transferred by
the ERNIC along with the door bells rang.
c. RDMA WRITE REQUEST:
- Example design has a predefined RDMA Write work Queue entry Packets. The
WQE Generator module rings the SQPI doorbell and sends the work queue entry
packets when requested by the ERNIC.
- The TX Checker Module checks the necessary fields of the RDMA write request
received.
- Upon successful validation of the request, the Packet Generator module sends
ACK packet to the ERNIC IP.
d. RDMA WRITE:
- For Initiator functionality, RDMA write packets are sent to the ERNIC module
which writes payload data to the RX path and the acknowledgment (ACK packet)
is received on the TX path.

ERNIC v4.0 67
PG332 December 2, 2022 www.xilinx.com
 Chapter 5: Example Design

- The ACK packet and data is checked through the checker modules.
e. RDMA READ:
- For initiator Functionality, RDMA Read packets are sent to ERNIC module which
reads the data from DDR and sends the read response on the TX path.
- The Read response packets are checked through checker module.

ERNIC v4.0 68
PG332 December 2, 2022 www.xilinx.com
 Chapter 6

ERNIC Software Flow

The ERNIC IP should be initialized and configured before it can be used for sending and
receiving RDMA traffic. This involves performing the following steps in sequence:

1. ERNIC IP global configuration.

2. Memory region registration (for RDMA client application which share the memory
region to remote hosts).
3. QP1 creation for sending and receiving RDMA connection management (CM) packets.
4. RC QP creation.
5. CQ creation.
6. Posting WQEs for RDMA operations.
7. Processing completions.
8. QP deletion.
9. QP fatal recovery.

ERNIC Global Configuration

The following registers should be configured for ERNIC global configuration.

• Configuration of error buffers queue. Allocate memory and configure the base address
to ERRBUFBA and queue depth and size to ERRBUFSZ registers. The register
ERRBUFWPTR is used by the hardware to indicate the SW about the new entries in the
ERROR buffer queue.
• Configure incoming packet error status queue. This queue gives the status of incoming
packets with errors. To configure this queue, allocate memory and write the base
address to IPKTERRQBA, queue depth and size to IPKTERRQSZ registers.
IPKTERRQWPTR register is used by the hardware to indicate the SW about the new
entries in the queue.
• Configure response error packet buffer. This involves writing memory base address to
RESP_ERR_PKT_BUF_BA, queue depth and size to RESP_ERR_BUF_SZ and a DDR address
to RESP_ERR_BUF_WRPTR for the hardware to indicate the SW about the response error
buffer writes happened.

ERNIC v4.0 69
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

• Enable interrupts by writing to INTEN register.

• Allocate memory for CQ and RQ doorbells for all the QPs to be created. The doorbell
memory for individual CQ and RQ are taken from offsets of this memory and
configured in the corresponding ERNIC registers during the QP creation (see the QP1
Creation).
• Configure source MAC address in the MACXADDLSB, MACXADDMSB registers.
• Configures source IP address. If the IP version is IPv6, then registers IPv6XADD1-4 are
written with interface's IPv6 address, otherwise IPv4 address is written to IPv4XADD
register.
• Configures number of QPs supported by the ERNIC design, UDP port and enables the
ERNIC by writing to XRNICCONF register.

QP1 Creation
QP1 is a special QP in the ERNIC IP. This QP is designated for exchanging the MAD packets
with remote hosts for establishing the connection for RC QPs. QP1 must be configured first
before creation of any other RC QP. See the InfiniBand Architecture Specification Volume 1,
Annex A16 RoCE and Annex 17 RoCE V2 [Ref 1] for more information on QP1. The following
steps are required to configure and enable QP1:

1. Allocate memory for queues (RQ, SQ, CQ), and program the respective base address
registers.

° RQ base address is written to RQBAi register. SQ base address is written to SQBAi

register and queue depths to QDEPTHi.

° CQ base address is written to CQBAi register.

° Configures each RQ buffer size by writing to QPCONFi.

2. Pick up memory for both SQ completion doorbell and RQ write pointers from the
pre-allocated doorbell memory pool as part of ERNIC initialization (see ERNIC Global
Configuration)

° SQ completion doorbell address to CQDBADDi register.

° RQ write pointer doorbell address to RQWPTRDBADDi register.

Memory Registration
Memory must be registered with ERNIC hardware before it is exchanged with remote hosts
for doing the RDMA operations (incoming RDMA READ and incoming RDMA WRITE). The
following steps are important to register a memory with ERNIC:

ERNIC v4.0 70
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

1. Allocate physical memory and create a virtual address mapping to it.

2. Create a protection domain number and write to PDPDNUM register at a free slot in the
MR table.
3. Write the virtual address of the memory region in VIRTADDRLSB and VIRTADDRMSB
registers at the same slot in PD table.
4. Write the physical address of the memory region to BUFBASEADDRLSB and
BUFBASEADDRMSB registers.
5. Create an R_KEY and configure it to BUFFERKEY register.
6. Write memory region length to WRRDBUFLEN register and access permission (REMOTE
READ or WRITE) to ACCESSDESC registers.

RC QP Creation
Once the QP1 is enabled, RC QPs can be created by exchanging the CM MAD packets on
QP1. See the “Communication Management” chapter in the InfiniBand Architecture
Specification Volume 1, Annex A16 RoCE and Annex 17 RoCE V2 [Ref 1] for details on CM
MAD packets. For creating any RC QP, the following steps need to be performed:

1. Allocate memory for queues (RQ, SQ, CQ), and program the respective base address
registers.

° RQ base address is written to RQBAi register, SQ base address is written to SQBAi

register and queue depths to QDEPTHi.

° CQ base address is written to CQBAi register.

° Configure the each RQ buffer size by writing to QPCONFi.

2. Pick up memory for both SQ completion doorbell and RQ write pointers from the
pre-allocated doorbell memory pool as part of ERNIC initialization.

° SQ completion doorbell address to CQDBADDi register.

° RQ write pointer doorbell address to RQWPTRDBADDi register.

3. When the QP state transitions from RESET to RTS during the connection establishment,
configure the following information:
a. Write remote host MAC address to MACDESADDLSBi and MACDESADDMSBi
registers.
b. Configure DESTQPCONFi with destination QP ID.
c. Configure IP address to IPDESADDR1i - IPDESADDR4i if IPv6 is being used otherwise,
write only IPDESADDR1i with IPv4 address. Configure the IP address version in the
QPCONFi.

ERNIC v4.0 71
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

d. Configure send queue PSN by writing to SQPSNi.

e. Configure last receive queue PSN by writing to LSTRQREQi
f. Configure PMTU by writing to QPCONFi register.
g. Configure the following by writing to TIMEOUTCONFi register.
- Configure RNR retry count and RNR retry timeout
- Configure retry timeout and retry timeout
h. PD number that the QP is associated with is configured by writing to Protection
Domain number register.

WQE SQ Posting
• The QP can now be used for posting work requests for RDMA operations. To do that
prepare the WQE in the format described in Table 2-1 and copy it to SQ memory.
• Ring the doorbell by incrementing SQPIi by 1 and writing the value to the SQPIi
register.

Receiving Incoming RDMA SEND Messages

Data received in the incoming SEND messages is copied to RQ buffers by ERNIC IP. The
software can retrieve this data by doing the following steps:

1. Read the RQWPTRDBADDi value. If there is a change from previous value, then process
the received data by reading the data from RQBAi+offset. This offset is calculated based
on RQ buffer size and the value in RQWPTRDBADDi.
2. After processing the received messages, increment RQCIi register value to indicate to
the hardware that buffer is consumed so it can be used for further incoming messages.

Processing WQE Completions

ERNIC IP indicates the completion of WQE posted to it, by incrementing the CQHEADi
register. Software should poll this register to check for any completions for the previously
posted WQEs.

ERNIC v4.0 72
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

Enabling QP HW Offload
The RC QPs can be enabled for HW handshake mode, where the external hardware
application can send and receive RDMA messages on this QP. The offloads the software
posting of RDMA messages to external hardware application. To enable the HW handshake
mode on a QP, the following steps need to be performed:

1. Enable software override by writing to XRNIC_ADV_CONF.

2. Reset the QP pointers by writing 0 to SQPIi, CQHEADi, and STATCURSQPTRi.
3. Configure remaining RQs to be processed by getting the difference between RQCIi and
RQWPTRDBADDi and write the value to STATRQPIDBi at lower 2B.
4. Enable HW offload by writing to QPCONFi.
5. Disable software override by writing to XRNIC_ADV_CONF.

QP Deletion
An RC can be deleted when it is no longer in use (either because of locally initiated
disconnect or from the remote peer). The following steps needs to be performed for
deleting the QP and removing its configuration:

1. Wait for SQ and outstanding queues to become empty. The status bits are in STATQPi
register.
2. Wait for all completions received for WQEs in SQ. This is done by checking SQPIi and
CQHEADi registers.
3. Enable the software override by writing to XRNIC_ADV_CONF register and disable the
QP by writing to QPCONFi register.
4. Reset the QP pointers by writing 0 to RQWPTRDBADDi, SQPIi, CQHEADi, RQCIi,
STATRQPIDBi, STATCURSQPTRi, SQPSNi, LSTRQREQi, STATMSN, and then disable the QP
and kept it in recovery mode by writing QPCONFi.
5. After resetting pointers, software override should be disabled by writing to
XRNIC_ADV_CONF.
6. After this, software should free memory allocated for SQ, RQ, and CQs.

ERNIC v4.0 73
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

QP Fatal Recovery
The following steps describe the recovery process for a QP that entered the fatal condition.
It describes how to clear the existing traffic on the QP and re-initialize it so that it can be
reused.

Steps to clear traffic on QP:

1. On detecting QP under the FATAL interrupt, read the IPKTERRQBA register to know the
Incoming Pkt Error Status Queue base address written by the driver.
2. Read this base address to know the QP FATAL status and decide which QP went into
FATAL (Bits[31:16]) and check QP FATAL status code (Bits[15:0]), see Table 2-4.
3. Stop pushing any further SQ PI doorbells.
4. Set the “QP under recovery” bit in the QPCONFi register to 1.
5. Read the STATQPi register to check “send Q empty” and “outstanding Q empty” bits to
become 1.
6. Poll the CQHEADi register to check its value is the same as SQPIi register.
7. Poll the RESPHNDSTS register for “sq pici db check en” —16 th bit to be set.
8. Set the QPCONFi register “QP enable” bit to 0 and “QP under recovery” bit to 1.

Steps to reinitialize the QP:

1. Poll the CQHEADi register to check its value is the same as SQPIi register.
2. Poll the RESPHNDSTS register for “sq pici db check en”—16 th bit to be set.
3. Set the “SW OVERWRIDE” bit in the XRNICADCONF register to 1.
4. Initialize the following QP registers to 0:

° STATRQPIDBi

° STATRQBUFCAi

° STATRQBUFCAMSBi

° RQCIi

° STATCURSQPTRi

° SQPIi

° SQPSNi

° LSTRQREQi

° STATMSN

ERNIC v4.0 74
PG332 December 2, 2022 www.xilinx.com
 Chapter 6: ERNIC Software Flow

° CQHEADi
5. Poll the CQHEADi register to check its value is 0.
6. Initialize the following register with the new value:

° SQPSNi

° LSTRQREQi
7. Initialize the following Ethernet side registers:

° MACDESADDMSBi

° MACDESADDLSBi

° IPDESADDR1i

° IPDESADDR2i

° IPDESADDR3i

° IPDESADDR4i
8. Re-configure the IP version in the QPCONFi.
9. Re-initialize the “RNR nack count” and “retry count” in the STATQPi register.
10. Set the ACCESSDESC register “access type” for that QP to 'b10.
11. Re-program the QPCONFi register by re-initializing fields like “RQ interrupt enable”, “CQ
interrupt enable”, “PMTU”, and “HW handshake disable”. Selectively enable “CQE write
enable” to debug error completions and re-initialize “RQ Buffer size”.
12. Re-program the QPADVCONFi register by randomizing “Traffic class”, “Time to live”, and
“Partition key”.

13. Set “QP EN” to 1 and “QP under recovery” to 0 in QPCONFi register.

14. Clear the “‘SW OVERWRIDE” bit in the XRNICADCONF register to 0.

ERNIC v4.0 75
PG332 December 2, 2022 www.xilinx.com
 Appendix A

Requesting ERNIC Support Questionnaire

The following is a questionnaire to be filled before requesting ERNIC support.

Table A‐1: ERNIC Support Questionnaire

Question Answer
ERNIC IP Configuration
ERNIC and Vivado release version
Hardware GUI parameter values
Hardware handshake mode enabled/disabled
ERNIC System Level Connectivity
ERNIC  ERNIC
ERNIC  Switch  ERNIC
ERNIC  Mellanox
ERNIC  Switch  Mellanox
ERNIC  BFM (Simulation)
Traffic Pattern
Opcode – RDMA_READ/RDMA_WRITE/RDMA_SEND/
IMMEDIATE/INVALIDATE/Mix of packets with different
Opcodes
Transfer size (DMA length and PMTU)
Traffic Direction
ERNIC sending packets (TX)
ERNIC receiving packets (RX)
Both
Issue Type
Hang
ERNIC is not sending ACKs
ERNIC is not processing the WQEs
ERNIC is not processing the incoming requests/
responses
Performance drop
Packet drop
Excessive retries

ERNIC v4.0 76
PG332 December 2, 2022 www.xilinx.com
 Appendix A: Requesting ERNIC Support Questionnaire

Table A‐1: ERNIC Support Questionnaire (Cont’d)

Question Answer
Unexpected fatal – Check if a Fatal response is sent by
the Host
Fatal recovery
FCS error – Check the behavior of packet FIFO between
ERNIC CMAC
Other
Congestion
PFC is enabled in ERNIC and CMAC?
What are the PFC counter values from CMAC?
Software Related Questions
Issue with reference design delivered by Xilinx?
Connection establishment is done via MAD packets?
Other
System block diagram
Details of any custom blocks connected to ERNIC
Register configuration details
Configuration register dump
Debug register dump
Simulation/Wireshark dump

ERNIC v4.0 77
PG332 December 2, 2022 www.xilinx.com
 Appendix B

Debugging
This appendix includes details about resources available on the Xilinx Support website and
debugging tools. Before reaching out to Xilinx Support, fill out these basic questionnaire
provided in Appendix A.

TIP: If the IP generation halts with an error, there might be a license issue. See License Checkers in
Chapter 1 for more details.

Debugging the IP/System

The ERNIC IP has a number of status registers implemented to allow for extensive debug in
case of a failure. These registers along with counter information from the initiator RNIC, for
example Mellanox, can provide vital clues to debug any failure. The debug tools available to
the system debugger working with ERNIC IP are:

• ERNIC complete register dump

• Promiscuous or packet sniffer mode counters (from the initiator RNIC),
• HW counters (from the initiator RNIC).

The steps to dump the different logs are explained in the following sections.

Complete Dump of the ERNIC Registers

If you are using smart cable, perform these steps:

1. Change the connect command directive with appropriate IP address of the Smart cable
using the following command:
$ connect -host <smartlink IP address>

2. Change the target using the following command:

target -set -nocase -filter {name =~ “<processor name>”}

3. Read the ERNIC IP register values using the following command:

$ mrd -force <ERNIC IP offset> <number of locations to read>

ERNIC v4.0 78
PG332 December 2, 2022 www.xilinx.com
 Appendix B: Debugging

Enable promisc Mode on the Initiator RNIC

1. Enable promisc mode on the initiator RNIC using the command:
$ ifconfig <interface name> promisc
$ netstat -i

2. At the end of the test, dump all the counter information from the initiator RNIC using
the command:
$ ethtool –S <RNIC Card name> > ethtool.log

HW Debug Counters for Mellanox RNIC

To access and dump hardware debug counters for Mellanox RNIC, use the command:

$ cd /sys/class/infiniband/mlx5_0/ports/1/hw_counters/
$ for file in `ls .`; do echo -n "${file}:"; cat $file; done

Note: For more information on hardware debug counters for Mellanox, see DOC-2572 on the
Mellanox community.

Some quick debug checks that you can do to ensure that the system is clean are listed here.

• Check the ethtool.log file for any link failures or CRC failures. Any non-zero value in
these two counters points towards an unstable link and can be the cause of failure. Two
snippets from the ethtool.log file are listed here.

Sample 1:

rx_wqe_err: 0
rx_mpwqe_filler: 0
rx_mpwqe_frag: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 0
rx_cqe_compress_pkts: 0
link_down_events_phy: 4
rx_out_of_buffer: 0
rx_vport_unicast_packets: 5
rx_vport_unicast_bytes: 422
tx_vport_unicast_packets: 10
tx_vport_unicast_bytes: 714
rx_vport_multicast_packets: 46
rx_vport_multicast_bytes: 7608

Sample 2:

rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 320587336
rx_packets_phy: 324924215
rx_crc_errors_phy: 0
tx_bytes_phy: 27657272230
rx_bytes_phy: 467673988652
tx_multicast_phy: 59

ERNIC v4.0 79
PG332 December 2, 2022 www.xilinx.com
 Appendix B: Debugging

tx_broadcast_phy: 32
rx_multicast_phy: 0
rx_broadcast_phy: 9
rx_in_range_len_errors_phy: 0

• Check the hw_counters on the initiator side. These counters give a picture of all fatal/
non-fatal errors seen by the initiator. A sample of the counters:
duplicate_request:0
implied_nak_seq_err:0
lifespan:10
local_ack_timeout_err:0
out_of_buffer:0
out_of_sequence:0
packet_seq_err:0
rnr_nak_retry_err:0
rx_atomic_requests:0
rx_read_requests:5
rx_write_requests:108307999

• Check the following register locations from the ERNIC register dump. The QP Status
(STATQPi) registers for all enabled QPs provide a status of the different QPs. Check if
the QP FATAL status is set to 1 in any of the QP status registers. For example, the QP
Status register for QP 5:
0x84020688: 30620601 • QP Fatal is set to 1

• If the QP is in FATAL state, no transactions are performed from this QP and the QP gets
disconnected. Bits[22:16] in the same register provide the last AETH syndrome received
from the initiator. In many cases the QP might go into FATAL state due to a NAK
syndrome received from the initiator. The NAK syndrome helps you understand the
failure being seen by the initiator RNIC card. In the above example, the AETH syndrome
of 0x62 indicates a “Remote Access Error” from the initiator. The decoding of the AETH
syndrome is provided in the Infiniband Architecture Specification Volume 1 (Release
1.2.1). For NAK code details in this specification, see Table 43: AETH Syndrome field and
Table 44: NAK Codes.
• Check the Incoming and outgoing NAK count registers ((INNACKPKTCNT) and
(OUTNACKPKTCNT)) at offset 0x134 and 0x138 for the number of incoming NAK
syndromes seen and number of NAK syndromes sent out. This number should normally
correlate with the number seen from the hw_counters seen at the initiator. In general
not all NAK codes are fatal. However, all NAK codes lead to retries and can lower the
overall performance of the system. A high number of NAK codes can be a cause of
concern.
• The total number of retries initiated by the target can be known from the Retry count
status register (RETRYCNTSTS) at offset 0x140. Normally this number will match with
the incoming NAK count. In case this number is more than the incoming NAK count
value, it might be due to timeouts. Timeouts happen when the responder (in this case,
the initiator RNIC) does not respond to a request in a given time. The timeout value is
configured in the Timeout Configuration register (TIMEOUTCONF). This timeout
interval is implemented as per the InfiniBand™ Architecture Specification Volume 1

ERNIC v4.0 80
PG332 December 2, 2022 www.xilinx.com
 Appendix B: Debugging

(Release 1.2.1) clause C9-141. It might be worthwhile to try and increase the timeout
interval and check if the number of retries is reduced.
• ERNIC register offset 0x6C (ERRBUFWPTR) indicates Error buffer write pointer. This
register gives the number of error packets received. Each error packet will be stored in
the address location provided in Error buffer base address (ERRBUFBA) register (offset
0x60). Each entry in this buffer will be given with error syndrome. See ERNIC RX Path
for details. The rows highlighted in yellow enlist the conditions that will cause the QP to
go into FATAL state.

ERNIC v4.0 81
PG332 December 2, 2022 www.xilinx.com
 Appendix C

Additional Resources and Legal Notices

Xilinx Resources
For support resources such as Answers, Documentation, Downloads, and Forums, see Xilinx
Support.

For a glossary of technical terms used in Xilinx documentation, see the Xilinx Glossary.

Documentation Navigator and Design Hubs

Xilinx ® Documentation Navigator provides access to Xilinx documents, videos, and support
resources, which you can filter and search to find information. To open the Xilinx
Documentation Navigator (DocNav):

• From the Vivado® IDE, select Help > Documentation and Tutorials.
• On Windows, select Start > All Programs > Xilinx Design Tools > DocNav.
• At the Linux command prompt, enter docnav.

Xilinx Design Hubs provide links to documentation organized by design tasks and other
topics, which you can use to learn key concepts and address frequently asked questions. To
access the Design Hubs:

• In the Xilinx Documentation Navigator, click the Design Hubs View tab.
• On the Xilinx website, see the Design Hubs page.
Note: For more information on Documentation Navigator, see the Documentation Navigator page
on the Xilinx website.

ERNIC v4.0 82
PG332 December 2, 2022 www.xilinx.com
 Appendix C: Additional Resources and Legal Notices

References
These documents provide supplemental material useful with this product guide:

1. InfiniBand Architecture Specification Volume 1, Annex A16 RoCE and Annex 17 RoCE V2
2. Vivado Design Suite User Guide: Designing IP Subsystems using IP Integrator (UG994)
3. Vivado Design Suite User Guide: Designing with IP (UG896)
4. Vivado Design Suite User Guide: Getting Started (UG910)
5. Vivado Design Suite User Guide: Logic Simulation (UG900)
6. Vivado Design Suite User Guide: Programming and Debugging (UG908)
7. Vivado Design Suite User Guide: Implementation (UG904)
8. Vivado Design Suite User Guide: AXI Reference Guide (UG1037)

Revision History
The following table shows the revision history for this document.

Date Version Revision

12/02/2022 4.0 • Updated Features.
• Updated Feature Summary.
• Updated Table 1-1.
• Updated Table 2-6.
• Updated Register Space.
• Updated Figure 2-9.
• Updated Table 2-7.
• Updated Table 3-1.
• Updated Table 3-2.
06/30/2021 3.1 Updated size to 256 in Table 3-1.

ERNIC v4.0 83
PG332 December 2, 2022 www.xilinx.com
 Appendix C: Additional Resources and Legal Notices

Date Version Revision

06/16/2021 3.1 • Added Versal ACAP.
• Added timing violation note in Features.
• Updated Figure 1-1.
• Added non-RDMA packet description in RX PKT Handler.
• Added retransmission description in Response Handler.
• Updated maximum DMA length and added Read response descriptions
in Unsupported Features.
• Updated Table 2-4.
• Added description for rx_pkt_hndler_ddr_m_axi* in Table 2-5.
• Updated description for 0x20088, 0x2013C, 0x20200 + ((i-1) x 0x0100),
and 0x202B0 + ((i-1) x 0x0100) in Table 2-7.
• Updated size in Table 3-1.
• Added QP Fatal Recovery.
• Added Appendix A, Requesting ERNIC Support Questionnaire.
02/26/2021 3.0 Updated Resource Utilization link.
01/21/2021 3.0 • Updated Priority flow control in Features.
• Added Navigating Content by Design Process section.
• Added Flow Control Manager in Feature Summary.
• Updated Figure 1-1.
• Added RDMA READ/WRITE request description in ERNIC RX Path.
• Updated Figure 2-6.
• Updated CMAC RX/TX descriptions and RoCE descriptions in Table 2-5.
• Updated Figure 2-9.
• Updated Table 2-7.
• Updated Figure 3-1.
• Updated address for RCV Q Buffer base address QPi (RQBAMSBi).
• Updated descriptions #2 and #3 in Complete Dump of the ERNIC
Registers.
06/10/2020 2.0 Added new chapter Chapter 6, ERNIC Software Flow
12/09/2019 1.0 Updated Table 2-5 and Table 3-2.
07/10/2019 1.0 Minor updates
12/05/2018 1.0 Initial Xilinx release.

ERNIC v4.0 84
PG332 December 2, 2022 www.xilinx.com
 Appendix C: Additional Resources and Legal Notices

Please Read: Important Legal Notices

The information disclosed to you hereunder (the “Materials”) is provided solely for the selection and use of Xilinx products. To the
maximum extent permitted by applicable law: (1) Materials are made available "AS IS" and with all faults, Xilinx hereby DISCLAIMS
ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY, NON-INFRINGEMENT, OR FITNESS FOR ANY PARTICULAR PURPOSE; and (2) Xilinx shall not be liable (whether
in contract or tort, including negligence, or under any other theory of liability) for any loss or damage of any kind or nature related
to, arising under, or in connection with, the Materials (including your use of the Materials), including for any direct, indirect, special,
incidental, or consequential loss or damage (including loss of data, profits, goodwill, or any type of loss or damage suffered as a
result of any action brought by a third party) even if such damage or loss was reasonably foreseeable or Xilinx had been advised
of the possibility of the same. Xilinx assumes no obligation to correct any errors contained in the Materials or to notify you of
updates to the Materials or to product specifications. You may not reproduce, modify, distribute, or publicly display the Materials
without prior written consent. Certain products are subject to the terms and conditions of Xilinx’s limited warranty, please refer to
Xilinx’s Terms of Sale which can be viewed at https://www.xilinx.com/legal.htm#tos; IP cores may be subject to warranty and
support terms contained in a license issued to you by Xilinx. Xilinx products are not designed or intended to be fail-safe or for use
in any application requiring fail-safe performance; you assume sole risk and liability for use of Xilinx products in such critical
applications, please refer to Xilinx’s Terms of Sale which can be viewed at https://www.xilinx.com/legal.htm#tos.
AUTOMOTIVE APPLICATIONS DISCLAIMER
AUTOMOTIVE PRODUCTS (IDENTIFIED AS “XA” IN THE PART NUMBER) ARE NOT WARRANTED FOR USE IN THE DEPLOYMENT OF
AIRBAGS OR FOR USE IN APPLICATIONS THAT AFFECT CONTROL OF A VEHICLE (“SAFETY APPLICATION”) UNLESS THERE IS A
SAFETY CONCEPT OR REDUNDANCY FEATURE CONSISTENT WITH THE ISO 26262 AUTOMOTIVE SAFETY STANDARD (“SAFETY
DESIGN”). CUSTOMER SHALL, PRIOR TO USING OR DISTRIBUTING ANY SYSTEMS THAT INCORPORATE PRODUCTS, THOROUGHLY
TEST SUCH SYSTEMS FOR SAFETY PURPOSES. USE OF PRODUCTS IN A SAFETY APPLICATION WITHOUT A SAFETY DESIGN IS FULLY
AT THE RISK OF CUSTOMER, SUBJECT ONLY TO APPLICABLE LAWS AND REGULATIONS GOVERNING LIMITATIONS ON PRODUCT
LIABILITY.
© Copyright 2012-2022, Advanced Micro Devices, Inc. Xilinx, the Xilinx logo, Alveo, Artix, Kintex, Kria, Spartan, Versal, Vitis, Virtex,
Vivado, Zynq, and other designated brands included herein are trademarks of Xilinx in the United States and other countries.
AMBA, AMBA Designer, Arm, ARM1176JZ-S, CoreSight, Cortex, PrimeCell, Mali, and MPCore are trademarks of Arm Limited in the
EU and other countries. The DisplayPort Icon is a trademark of the Video Electronics Standards Association, registered in the U.S.
and other countries. All other trademarks are the property of their respective owners.

ERNIC v4.0 85
PG332 December 2, 2022 www.xilinx.com