Scalable Distribution in Dual-SmartNICs, Dual-NUMA Systems

NUMA-aware stream assignment

This NTPL and ntservice.ini configuration demonstrates how to independently capture traffic from two SmartNICs installed in a dual-NUMA system. Each SmartNIC supports a maximum of 256 or 128 RX streams, for a total of 512 or 256 streams that are local to its associated NUMA node, optimizing memory locality and system performance.

Configure RX host buffers in the ntservice.ini file as follows:

[Adapter0]
…
HostBuffersRx = [128,16,0]
…

[Adapter1]
…
HostBuffersRx = [128,16,1]
…

The first parameter in HostBuffersRx sets the number of host RX buffers to 128 for each SmartNIC.
The third parameter in HostBuffersRx = [128,16,<NUMA>] explicitly sets the NUMA node for memory allocation.
- Adapter0 is associated with NUMA node 0.
- Adapter1 is associated with NUMA node 1.

Setup[NUMANode=0] = StreamId == (0..127)
Setup[NUMANode=1] = StreamId == (128..255)

Assign[StreamID=(0..127)] = Port==(0..1)
Assign[StreamID=(128..255)] = Port==(2..3)

The Setup commands bind streams 0–127 to NUMA node 0 and streams 128–255 to NUMA node 1.
The Assign command maps:
- Ports 0 and 1 (belonging to Adapter0) to streams 0–127.
- Ports 2 and 3 (belonging to Adapter1) to streams 128–255.

This setup enables efficient use of all available streams while maintaining NUMA affinity for high-throughput capture on systems with two SmartNICs. See ntservice.ini file in DN-0449.

Note: To use more than 256 network streams, the MaxNetStreams and MaxStreamIds parameters should be configured. The value of MaxNetStreams must be chosen based on the number of SmartNICs installed in the server and the design of the application. See Setting network stream limits.