• Amount of host buffers
  Performance is very dependant on the number of host buffers defined. If the configured number of host buffers is higher than the actual number required, performance will suffer because even though there is no traffic assigned to the unused host buffers they still need to be serviced which will consume CPU and PCIe bandwidth.
  In order to get an idea on the amount of required host buffers, one can use the Profiling tool located in the tools package. The tool is able to show how many host buffers are used and by which applications/streams.
• NUMA node
  PCIe performance can be limited depending on the server layout. If the server has a configuration with multiple CPUs and with multiple IOH (I/O hub) one needs to pay attention to the NUMA nodes and assign host buffers to an adapter to the NUMA node with the shorted QPI (QuickPath Interconnect) distance.
  

  One or multiple IOH
  The system PCIe read/write to each NUMA node is measured when NtService is stating. The measurement is made by letting the adapter perform reads/writes to host memory and measure how fast it can do it.
  
  Below is an extract from ntlog on a system with 2 NUMA nodes but only one IOH.

          # /opt/napatech3/bin/ntlog | cut -d- -f2
          ....
          PCIe settings:
          Supported max payload of 256 bytes. Negotiated 256 bytes
             PCIgen in use 2. Adapter supports PCIgen 2.
             Adapter support 8 lanes @ 5.0 GT/s. Link negotiated to 8 lanes @ 5.0 GT/s.
           PCIe measurements on NUMA node 0
             RX only max expected throughput is: 29041 Mbps
             TX only max expected throughput is: 28259 Mbps
             RX combined max expected throughput is: 27406 Mbps
             TX combined max expected throughput is: 27821 Mbps
           PCIe measurements on NUMA node 1
             RX only max expected throughput is: 29057 Mbps
             TX only max expected throughput is: 28259 Mbps
             RX combined max expected throughput is: 27399 Mbps
             TX combined max expected throughput is: 27804 Mbps
          ....

  The extract below is the same dump but on a system with 2 IOH and 2 two NUMA nodes.

          # /opt/napatech3/bin/ntlog | cut -d- -f2
          ....
          PCIe settings:
          Supported max payload of 256 bytes. Negotiated 256 bytes
             PCIgen in use 2. Adapter supports PCIgen 2.
             Adapter support 8 lanes @ 5.0 GT/s. Link negotiated to 8 lanes @ 5.0 GT/s.
           PCIe measurements on NUMA node 0
             RX only max expected throughput is: 29041 Mbps
             TX only max expected throughput is: 28259 Mbps
             RX combined max expected throughput is: 27406 Mbps
             TX combined max expected throughput is: 27821 Mbps
           PCIe measurements on NUMA node 1
             RX only max expected throughput is: 19057 Mbps
             TX only max expected throughput is: 18259 Mbps
             RX combined max expected throughput is: 17399 Mbps
             TX combined max expected throughput is: 17804 Mbps
          ....

• Intel Turbo Boost
  Availability and frequency upside of Turbo Boost state depends upon several factors including, but not limited to, the following:
  
  • Type of workload
    
  • Number of active cores
    
  • Estimated current consumption
    
  • Estimated power consumption
    
  • Processor temperature
    The operation is dependent on headroom available in one or more cores. The amount of time the system spends in Turbo Boost mode depends on workload, environment, platform design, and overall system configuration.
    Maximum frequency for processor function can't be specified. Maximum frequency is automatic and dependent on working conditions.
    Due to varying power characteristics, some parts with Turbo Boost may not achieve maximum turbo frequencies when running heavy workloads and using multiple cores concurrently.
    The less total power the CPU is using, the higher it can set the clocks in turbo. The more cores that are idle the better performance with Turbo Boost on one single core.
    
• Network macros
  The Network macros are by default able to run on all supported packet descriptors. Running this may might not be the most optimal way because each of the macros needs to determine the descriptor format and issue an if-then-else until the correct version is located. Below is an example of a network macro implementation to illustrate the if-then-else case:
  #define NT_NET_GET_PKT_HASH(_n_) _NT_NET_GET_PKT_HASH(_h_)

  ...

  #if defined(_NTAPI_EXTDESCR_7_)

  #define NT_NET_GET_PKT_HASH(_h_) _NT_NET_GET_PKT_HASH_EXT7(_h_)

  #elseif defined(_NTAPI_EXTDESCR_8_)

  #define NT_NET_GET_PKT_HASH(_h_) _NT_NET_GET_PKT_HASH_EXT8(_h_)

  #elseif defined(_NTAPI_EXTDESCR_9_)

  #define NT_NET_GET_PKT_HASH(_h_) _NT_NET_GET_PKT_HASH_EXT9(_h_)

  #else

  #define NT_NET_GET_PKT_HASH(_h_) ((NT_NET_GET_PKT_DESCRIPTOR_FORMAT(_h_)==7)?

  _NT_NET_GET_PKT_HASH_EXT7(_h_):NT_NET_GET_PKT_DESCRIPTOR_FORMAT(_h_)==8?

  _NT_NET_GET_PKT_HASH_EXT8(_h_):NT_NET_GET_PKT_DESCRIPTOR_FORMAT(_h_)==9?

  _NT_NET_GET_PKT_HASH_EXT9(_h_):assert(0))

  #endif
  Two options exist to prevent the macros performing if-then-else:
  1. Define NTAPI_EXTDESCR? to match the extended descriptor in ntservice.ini. Macros not available for the choosen descriptor type will return -1.
  2. Use NT_NET_GET_PKT_DESCRIPTOR_FORMAT() to detect the format and use the format descriptors directly
     if(NT_NET_GET_PKT_DESCRIPTOR_FORMAT(hNetBuf)==7) {

     hash=_NT_NET_GET_PKT_HASH_EXT7(hNetBuf);

     hashType=_NT_NET_GET_PKT_HASH_TYPE_EXT7(hNetBuf);

     }

      [Adapter0]
      AdapterType=NT4E
      BusId=00:0a:00:00
      HostBuffersTx=[0,16,0] # 0 TX host buffers, disable the TX host buffers for adapter 0

      [Adapter0]
      AdapterType=NT4E
      BusId=00:0a:00:00
      HostBuffersRx=[0,16,0] # 0 RX host buffers, disable the RX host buffers for adapter 0