profiling
The Profiling tool is used to gain an overview of how host buffers are utilized and where data is currently located.
Syntax
profiling [-h | --help]
| Command | Description |
|---|---|
-h, --help | Show help and exit. |
The tool is divided into three windows:
Host Buffers
The Host buffers window gives a complete overview of all host buffers used by the NSS driver.
The different buffers can be selected by moving the cursor up or down using the arrow up and arrow down keys.
Pressing Enter or right arrow on a host buffer presents a Host Buffers window with further details. To exit this window press Esc or left arrow.
Note: The number of host buffers is defined in the ntservice.ini File.
┌───────────────────────────────────────────────────────────────────────────┐ │ Hb: Bytes: Pkts: Drop pkts: Buf.: #: ID: │ │ ------------------------------------------------------------------------ │ │ 0 0:3 TX - - - - 1 - │ │ 1 0:2 TX - - - - 1 - │ │ 2 0:1 TX - - - - 1 - │ │ 3 0:0 TX - - - - 1 - │ │ 4 0:7 RX 4AB7C480 12ADF12 0 99% 1 1 │ │ 5 0:6 RX 47D9AF80 11F66BE 0 99% 1 2 │ │ 6 0:5 RX 4C034640 1300D19 0 99% 1 3 │ │ 7 0:4 RX D4276C0 3509DB 0 0% 0 4 │ │ 8 0:3 RX 34DC9F80 D3727E 0 99% 3 5 │ │ 9 0:2 RX 0 0 0 0% 0 - │ │ 10 0:1 RX 0 0 0 0% 0 - │ │ 11 0:0 RX 0 0 0 0% 0 - │ │ 12 1:3 TX - - - - 1 - │ │ 13 1:2 TX - - - - 1 - │ │ 14 1:1 TX - - - - 1 - │ │ 15 1:0 TX - - - - 1 - │ │ 16 1:7 RX 499D9C40k 1267671 0 99% 1 5 │ │ 17 1:6 RX 48A53AC0k 12294EB 0 99% 1 1 │ └───────────────────────────────────────────────────────────────────────────┘
| Key | Action |
| Hb <index> <adapter>:<host buffer> | index - the host buffer index, adapter - the adapter number, host buffer - the host buffer number on the adapter |
| Bytes | The number of bytes received in the host buffer Not valid for TX host buffers |
| Pkts | The number of packets received in the host buffer Not valid for TX host buffers |
| Drop pkts | The number of dropped packets in the host buffer Not valid for TX host buffers |
| Buf | The percent of used adapter memory Not valid for TX host buffers |
| # | Number of streams/applications attached to the host buffer |
| ID | The stream ID attached to the host buffer |
Buf. - Percentage of used adapter memory
This shows the percentage of used adapter memory. One must pay attention to this value.
When the value is close to 100 %, it means that the adapter has used all the buffer memory.
This happens if the adapter is not able to transfer the received data to the host buffers on the PC
and is mainly caused by a slow application not being able to release the data fast enough.
Packet loss will probably occur.
Hb <index> <adapter>:<hostbuffer>
This shows the number of host buffers allocated in the ntservice.ini File for each adapter. This means that
the number can vary from adapter to adapter depending of the definitions in the ntservice.ini File.
Host Buffer Info (RX)
When an RX host buffer is seleced in the main window, this sub window is shown.
┌─────────────────────────────────────────────────────────────────────────────┐ │ Bytes: Pkts: Drop pkts: Buf.: #: ID: │ │ 0 0 0 0% 1 1 │ │ RX Host buffer info for host buffer 4 │ │ Attached to adapter 0. Size 16384k, NUMA node: 0. Egress port: None │ │ Onboard buffer size/crc errors: 255M / 0x00000000 │ │ Dequeued: 2048 Enqueued driver: 0 Enqueued adapter: 14336 │ │ │ │ Connected streams │ │ PID: Allowance Pkts dropped: Octets dropped: │ │ cur Level │ │ -------------------------------------------------------------------------- | │ 2272 0 16384 0 0 │ └─────────────────────────────────────────────────────────────────────────────┘
| Key | Action |
| Bytes, Pkts, DropPkts, Buf, #, ID | See Host Buffers. The values are extracts from that window. |
| Size | Size of the host buffer |
| NUMA node | The NUMA node on which the host buffer is located |
| Egress port | The Egress port used |
| Onboard buffer size | Adapter buffer size - the size of the SDRAM mounted on the adapter |
| crc errors | Number of errors found by the adapter |
| Dequeued | Amount of host buffer data owned by the application |
| Enqueued driver | Amount of host buffer data owned by the driver |
| Enqueued adapter | Amount of host buffer data owned by the adapter |
| Dropped packets | Number of packets dropped (lost) |
| Dropped bytes | Number of bytes dropped (lost) |
| Connected streams | Streams connected to the host buffer
|
Dequeued - Enqueued Driver - Enqueued Adapter
These values show where the host buffer memory currently is placed. This is used the find the
bottleneck in the data reception.
- Dequeued
This shows the amount of host buffer data which is currently in use or available to the application/stream. If this
value is increasing it indicates that the application is not able to handle and release data fast enough and can cause
the adapter to run out of host buffer space and therefore store packets in its on-board memory if available. Eventually,
when the on-board storage is filled, packet drops will occur.
- Enqueued driver
This shows the amount of host buffer data currently held by the driver. This is the data that can be provided to the
adapter either for TX or RX. - Enqueued adapter
This shows the amount of host buffer data currently queued in the adapter.
Normally the data should change among the three states but as the update rate of the profiling
tool is limited it is not always visible. One should only pay attention to these values if a
packet loss occurs.
- Note
- Even though there is no traffic, the numbers will update because keep alive packets are received
by the adapter. A keep alive packet is a time stamp sync. packet that is used when merging between multiple
host buffers and these are always enabled and will be interleaved with normal data traffic. The keep alive packets
are discarded before returning data via the network stream.
Connected Streams
These values show the streams connected to this host buffer and how much of the host buffer each stream is using or has pending.
The application/stream that has the highest memory use will probably be the application
causing the bottleneck as it is too slow to release the packets. In this way the application will lock
the host buffer for all other applications as the memory can only be released to the adapter when all
applications connected to the host buffer have read and released the memory.
Host Buffer Info (TX)
When a TX host buffer is selected in the main window, this sub window is shown.
┌───────────────────────────────────────────────────────────────────────────────┐ │ Bytes: Pkts: Drop pkts: Buf.: #: ID: │ │ Adapter: 0 Hostbuffer #: 3 Size: 8192k NUMA node: 0. Egress port: 0 │ │ Dequeued: 15 Enqueued driver: 0 Enqueued adapter: 8176 │ │ │ │ Connected stream: PID=2088 │ │ │ │ │ └───────────────────────────────────────────────────────────────────────────────┘
| Key | Action |
| Adapter | Adapter that the host buffer belongs to |
| Hostbuffer # | Host buffer number of the host buffer assigned to the adapter |
| Size | Size of the host buffer |
| NUMA node | The NUMA node on which the host buffer is located |
| Egress port | The egress port the host buffer is assigned to |
| Dequeued | Amount of host buffer data owned by the application |
| Enqueued driver | Amount of host buffer data owned by the driver |
| Enqueued adapter | Amount of host buffer data owned by the adapter |
| Connected stream | Process ID of the application connected to the host buffer |
ntservice.ini File
This is a short example of how host buffers are defined in the ntservice.ini file.
[adapterX]
HostBuffersRx=[8,16,0]
HostBuffersTx=[4,16,0]
The example shows that there are 8 RX host buffers defined with a size of 16 MBytes from NUMA node 0
and 4 TX host buffers with a size of 8 MBytes from NUMA node 0.
In the main window the host buffers will be shown as:
X:3 TX
X:2 TX
X:1 TX
X:0 TX
X:8 RX
X:7 RX
X:6 RX
.........
X:0 RX
A further description of the ntservice.ini file can be found in the ntservice.ini section of the Configuration file.
Receive Debugging and Optimising
- Check Dequeued - Enqueued driver - Enqueued adapter - If data is placed at Dequeued this means that the application is too slow to handle the received data.
It is necessary to optimize the application.
- Buf - If the value is close to 99%, this means as above that the application is too slow.
- Connected streams - If more than one application is connected to the host buffer (sharing), this shows the memory used
by each application. If packet loss occurs, the application with the highest memory usage is the
one causing the packet loss.
The reception can be optimised by changing the number and size of the host buffers.
Stream List Window
The Stream List window shows the open streams in the system.
┌──────────────────────────────────────────────────────────────────────────────┐ │ Open streams: 3 (2 within this application but not shown below) │ │ id: Name Type PID Application name │ │ -------------------------------------------------------------------------- │ │ 2 Throughput network 2272 throughput │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ -------------------------------------------------------------------------- │ │ Direction: Receive │ │ Interface type: Packet │ │ Stream Id: 1 │ │ │ │ │ │ │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────────────────────┘
| Key | Action |
| id | The stream handle # |
| Name | The name of the stream provided when opened |
| Type | The type of stream: Network, Statistics, Info, Config, Event |
| PID | The Process ID of the application owning the stream |
| Application name | The name of the application owning the stream |
Stream ID List Window
The Stream ID window shows the stream IDs created either via NTPL or via NT_NetRxOpen().
┌──────────────────────────────────────────────────────────────────────────────┐ │ Created stream ids: │ │ id: state #host buffers: #streams: │ │ -------------------------------------------------------------------------- │ │ 1 Active 1 1 │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────────────────────┘
| Key | Action |
| id | The stream ID |
| state | The state of the stream |
| #host buffers | The number of host buffers in use by the stream ID (merging is applied if more than one is in use) |
| #streams | The number of streams using the stream ID (data sharing if more than one is in use) |