Configuring the IPU

Getting Started with Napatech Link-Storage™ Software

Platform
Napatech IPU
Content Type
Getting Started
Capture Software Version
Link-Storage™ Software 1.0

Use this information to configure the Napatech SPDK on the IPU for attaching NVMe devices of the target server.

Before you begin

Make sure that you have:

About this task

Note: The following prompts are used to indicate which part of the system to run the provided commands on.
  • soc#: The SoC on the IPU.
  • host#: The server where the IPU is installed.
  • target#: The remote server with storage disks.

Procedure

  1. On the IPU, load the drivers.
    soc# modprobe virtio_net
    soc# modprobe uio
    soc# rmmod ifc_uio.ko 2>/dev/null
    soc# cd /opt/ipu_workload_nvme/src/
    soc# insmod ipu_mngmnt_tools/software/csc_lek-0.0.0.5/driver/kmod/ifc_uio.ko
  2. Initialize the packet processor (PP) switch.
    soc# export CSC_MNGMT_PATH=/opt/ipu_workload_nvme/src/ipu_mngmnt_tools/software/management 
    soc# /opt/ipu_workload_nvme/src/ipu_mngmnt_tools/ppfastpath.sh
    The ppfastpath.sh script contains the following commands.
    cd $CSC_MNGMT_PATH
    ./csc_mngmt --dev 15:00.0 --flush --all
    ./csc_mngmt --dev 15:00.0 --sync
    ./csc_mngmt --dev 15:00.0 --ppcfg fastpath
    An output example:
    PP Fastpath Configuration was SUCCESSFUL!
    Note: If the following error messages are generated, reboot the IPU.
    ** Error: Unable to parse file '/var/csc-config.json'. Error code: 1003. Regenerating...
    *** Error: Invalid Action Group Entry for Sys IF 0 Broadcast whilst trying to configure PP fastpath default config
    *** Error: Invalid Action Group Entry for Sys IF 1 Broadcast whilst trying to configure PP fastpath default config
    *** Error: Invalid Action Group Entry for Sys IF 0 Broadcast whilst trying to configure PP fastpath default config
    *** Error: Invalid Action Group Entry for Sys IF 1 Broadcast whilst trying to configure PP fastpath default config
    PP Fastpath Configuration contained at least one ERROR
    After reboot, the drivers must be loaded again as described in Step 1 and run the ppfastpath.sh script again.
    soc# reboot
  3. On the IPU, add destination network interfaces to the PP lookup table.
    soc# cd $CSC_MNGMT_PATH
    soc# ./csc_mngmt  --dev 15:00.0 --lkup --add --lkup-tag 4 \
    --dmac <target_port0_MAC> --route-dest LINE:0:0
    soc# ./csc_mngmt  --dev 15:00.0 --lkup --add --lkup-tag 4 \
    --dmac <target_port1_MAC> --route-dest LINE:1:0
    where:
    • target_port0_MAC is the MAC address of the NIC on the target server, which is connected to port 0 of the IPU.
    • target_port1_MAC is the MAC address of the NIC on the target server, which is connected to port 1 of the IPU.
    For example:
    cd $CSC_MNGMT_PATH
    soc# ./csc_mngmt  --dev 15:00.0 --lkup --add --lkup-tag 4 \
    --dmac 64:9d:99:ff:ed:dc --route-dest LINE:0:0
    soc# ./csc_mngmt  --dev 15:00.0 --lkup --add --lkup-tag 4 \
    --dmac 64:9d:99:ff:ed:dd --route-dest LINE:1:0
  4. On the IPU, configure IP addresses of two interfaces.
    soc# ip link set dev ens6f1 down
    soc# ip a a 172.168.1.1/24 dev ens6f1
    soc# ip link set dev ens6f1 up
    
    soc# ip link set dev ens6f2 down
    soc# ip a a 172.168.2.1/24 dev ens6f2
    soc# ip link set dev ens6f2 up
    Note: ens6f4 must not be used.
  5. On the IPU, verify the network connections between the IPU and the target server.
    soc# ping 172.168.1.2
    If it fails to ping to the target server, make sure that:
    • All network connections are established.
    • Pluggable modules are securely inserted.
    • Cables are properly connected.
    • Related devices are functioning correctly without defects.
  6. On the IPU, enable the drivers.
    soc# /sbin/rmmod vfio-pci
    soc# /sbin/rmmod vfio_iommu_type1
    soc# /sbin/rmmod vfio
    soc# /sbin/modprobe vfio-pci
    soc# chmod a+x /dev/vfio
  7. Bind the management PCI function of the IPU to the vfio-pci driver.
    The PCIe bus ID of the management PCI function on the IPU is 0000:15:00.4.
    soc# cd /opt/ipu_workload_nvme/src/
    soc# ipu_spdk/dpdk/usertools/dpdk-devbind.py -b vfio-pci 0000:15:00.4
  8. On the IPU, start bsc_tgt application. Keep the terminal open.
    soc# ipu_spdk/test/external_code/bsc_tgt/bsc_tgt -s 5G \
    --base-virtaddr 0x2000000000 -m 0x0f
    where:
    • -s 5G: Specifies the size of the shared memory used by the application to 5 Gbytes.
    • --base-virtaddr 0x2000000000: Sets the starting address for the application to allocate the memory.
    • -m 0x0f: Configures the CPU affinity mask for the application. The value 0x0f indicates the application to run on CPUs 0, 1, 2, and 3.
    An output example:
    [2024-02-29 05:28:46.370746] Starting SPDK v22.01.3-pre git sha1 071e9fa55 / DPDK 21.11.2 initialization...
    [2024-02-29 05:28:46.370982] [ DPDK EAL parameters: [2024-02-29 05:28:46.371010] virtio_blk_net [2024-02-29 05:28:46.371026] --no-shconf [2024-02-29 05:28:46.371041] -c 0x0f [2024-02-29 05:28:46.371056] -m 5120 [2024-02-29 05:28:46.371070] --log-level=lib.eal:6 [2024-02-29 05:28:46.371092] --log-level=lib.cryptodev:5 [2024-02-29 05:28:46.371109] --log-level=user1:6 [2024-02-29 05:28:46.371126] --base-virtaddr=0x2000000000 [2024-02-29 05:28:46.371143] --match-allocations [2024-02-29 05:28:46.371159] --file-prefix=spdk_pid3801 [2024-02-29 05:28:46.371173] ]
    EAL: No available 2048 kB hugepages reported
    TELEMETRY: No legacy callbacks, legacy socket not created
    [2024-02-29 05:28:47.433020] app.c: 601:spdk_app_start: *NOTICE*: Total cores available: 4
    [2024-02-29 05:28:47.573122] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 1
    [2024-02-29 05:28:47.573346] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 3
    [2024-02-29 05:28:47.573347] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 0
    [2024-02-29 05:28:47.573237] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 2
    [2024-02-29 05:28:47.573931] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to 
  9. Open the management interface.
    soc# chmod 777 /var/tmp/spdk.sock
    soc# ./ipu_spdk/scripts/rpc.py bsc_open_mgm_interface "0000:15:00.0"
    The physical PCIe bus ID of the IPU must be specified as shown in this command.
  10. Attach an NVMe controller to NVMe devices of the target server.
    For example:
    soc# for (( i=0; i<16; i++ ))
    do
        n="nqn.2016-06.io.spdk:cnode"$((i))
        port=$((4420 + i))
        echo "attach controller NVMe"$i" to TCP node "$n
       ./ipu_spdk/scripts/rpc.py bdev_nvme_attach_controller -b NVMe$i -t tcp \
     -a 172.168.1.2 -s $port -f ipv4 -n $n
    done
    where:
    • -b NVMe$i: Specifies the name of the NVMe block device to which the controller will be attached. Names NVMe0 to NVMe15 are used.
    • -t tcp: Sets the transport protocol to TCP.
    • -a 172.168.1.2: Specifies the IP address of the target server.
    • -s $port: Specifies the port number. Port numbers 4420 to 4435 are configured on the target server.
    • -f ipv4: Specifies the IP protocol version as IPv4.
    • -n $n: Specifies the NQN (NVMe qualified name) of the controller. -n nqn.2016-06.io.spdk:cnode0 to -n nqn.2016-06.io.spdk:cnode15 are configured on the target server.
    An output example:
    attach controller NVMe0 to TCP node nqn.2016-06.io.spdk:cnode0
    NVMe0n1
    attach controller NVMe1 to TCP node nqn.2016-06.io.spdk:cnode1
    NVMe1n1
    attach controller NVMe2 to TCP node nqn.2016-06.io.spdk:cnode2
    NVMe2n1
    attach controller NVMe3 to TCP node nqn.2016-06.io.spdk:cnode3
    NVMe3n1
    attach controller NVMe4 to TCP node nqn.2016-06.io.spdk:cnode4
    NVMe4n1
    attach controller NVMe5 to TCP node nqn.2016-06.io.spdk:cnode5
    NVMe5n1
    attach controller NVMe6 to TCP node nqn.2016-06.io.spdk:cnode6
    NVMe6n1
    attach controller NVMe7 to TCP node nqn.2016-06.io.spdk:cnode7
    NVMe7n1
    attach controller NVMe8 to TCP node nqn.2016-06.io.spdk:cnode8
    NVMe8n1
    attach controller NVMe9 to TCP node nqn.2016-06.io.spdk:cnode9
    NVMe9n1
    attach controller NVMe10 to TCP node nqn.2016-06.io.spdk:cnode10
    NVMe10n1
    attach controller NVMe11 to TCP node nqn.2016-06.io.spdk:cnode11
    NVMe11n1
    attach controller NVMe12 to TCP node nqn.2016-06.io.spdk:cnode12
    NVMe12n1
    attach controller NVMe13 to TCP node nqn.2016-06.io.spdk:cnode13
    NVMe13n1
    attach controller NVMe14 to TCP node nqn.2016-06.io.spdk:cnode14
    NVMe14n1
    attach controller NVMe15 to TCP node nqn.2016-06.io.spdk:cnode15
    NVMe15n1
  11. On the IPU, verify established connections.
    soc# netstat | grep ESTABLISHED
    An output example:
    tcp  0  0 target:4433  172.168.1.1:53920  ESTABLISHED
    tcp  0  0 target:4422  172.168.1.1:59064  ESTABLISHED
    tcp  0  0 target:4430  172.168.1.1:34978  ESTABLISHED
    tcp  0  0 target:4434  172.168.1.1:53888  ESTABLISHED
    tcp  0  0 target:4435  172.168.1.1:39038  ESTABLISHED
    tcp  0  0 target:4426  172.168.1.1:48662  ESTABLISHED
    tcp  0  0 target:4427  172.168.1.1:41692  ESTABLISHED
    tcp  0  0 target:4420  172.168.1.1:54728  ESTABLISHED
    tcp  0  0 target:4428  172.168.1.1:36676  ESTABLISHED
    tcp  0  0 target:4432  172.168.1.1:51988  ESTABLISHED
    tcp  0  0 target:4421  172.168.1.1:46412  ESTABLISHED
    tcp  0  0 target:4431  172.168.1.1:47712  ESTABLISHED
    tcp  0  0 target:4429  172.168.1.1:48362  ESTABLISHED
    tcp  0  0 target:4424  172.168.1.1:46612  ESTABLISHED
    tcp  0  0 target:4423  172.168.1.1:59912  ESTABLISHED
    tcp  0  0 target:4425  172.168.1.1:47218  ESTABLISHED
    …
    …
    16 connections are established in this example.
  12. Construct a virtio block device.
    soc# ./ipu_spdk/scripts/rpc.py virtio_blk_net_construct bndev0 0000:15:00.4
    This constructs a virtio block device with the name bndev0 using PCI bus ID 0000:15:00.4.
  13. On the IPU, map NVMe devices to a virtio block device.
    soc# for (( i=0; i<16; i++ ))
    do
        d="NVMe${i}n1"
        echo "map "$d" to bndev0"
        ./ipu_spdk/scripts/rpc.py virtio_blk_net_map_bdev bndev0 $((15-i)) $d
    done
    where:
    • bndev0: Specifies the virtio network device to which the block device will be mapped.
    • $((15-i)): Specifies the queue index.
    • $d: Specifies the name of the block device to be mapped. In this case, it's named NVMe0n1 to NVMe15n1.
    An output example:
    map NVMe0n1 to bndev0
    map NVMe1n1 to bndev0
    map NVMe2n1 to bndev0
    map NVMe3n1 to bndev0
    map NVMe4n1 to bndev0
    map NVMe5n1 to bndev0
    map NVMe6n1 to bndev0
    map NVMe7n1 to bndev0
    map NVMe8n1 to bndev0
    map NVMe9n1 to bndev0
    map NVMe10n1 to bndev0
    map NVMe11n1 to bndev0
    map NVMe12n1 to bndev0
    map NVMe13n1 to bndev0
    map NVMe14n1 to bndev0
    map NVMe15n1 to bndev0