Activate Virtual Functions

Getting Started with Napatech Link-Virtualization™ Software

Platform
Napatech SmartNIC
Content Type
Getting Started Guide
Capture Software Version
Link-Virtualization™ Software 4.4

About this task

Virtual functions must be enabled after each system restart.

Procedure

  1. Load the vfio module and run the ntload.sh command.
    modprobe vfio-pci
    /opt/napatech3/bin/ntload.sh
    lsmod | grep vfio
    An output example:
    vfio_pci               41993  0
    vfio_iommu_type1       22440  0
    vfio                   32657  3 vfio_iommu_type1,vfio_pci
    irqbypass              13503  2 kvm,vfio_pci
    
  2. Activate a number of virtual functions.
    DEV=$(lspci -Dd 18f4: | cut -d' ' -f1 | head -1)
    NUMVF=5
    echo $NUMVF > /sys/bus/pci/devices/$DEV/sriov_numvfs
    cat /sys/bus/pci/devices/$DEV/sriov_numvfs
    where:

    $NUMVF represents the number of virtual channels to VMs, plus one for OVS. Thus using NUMVF=3 will enable 3 channels for VMs. The current maximum supported Virtual Functions count is 128.

    Note: 33.00 or newer version FPGA images support SR-IOV capabilities. If the FPGA image of the SmartNIC has been switched from an older version to 33.00 or newer version, you must reboot the server to enumerate SR-IOV capabilities.
  3. If step 2 fails with the following error, it is most likely because the Link-Virtualization™ Software driver is not loaded.
    echo $NUMVF > /sys/bus/pci/devices/$DEV/sriov_numvfs
    -bash: echo: write error: No such file or directory
    To check whether the Link-Virtualization™ Software driver is loaded, run:
    lsmod |grep nt3
    Load the driver if it is not loaded.
     /opt/napatech3/bin/ntload.sh
    Loading nt3gd driver                                        [Done]
    Creating driver device file                                 [Done]
    Loading nt3gd_netdev driver                                 [Done]
    Creating driver device file                                 [Done]
  4. Step 2 may fail with the following error in some old servers or some servers with old BIOS versions.
    echo $NUMVF > /sys/bus/pci/devices/$DEV/sriov_numvfs
    -bash: echo: write error: Cannot allocate memory
    You will find the following error messages in /var/log/dmesg:
    nt3gd 0000:03:00.0: sriov configure (res=3)
    nt3gd 0000:03:00.0: not enough MMIO resources for SR-IOV
    nt3gd 0000:03:00.0: sriov configure (res=-12)
    It is because the BIOS is not providing enough MMIO space for VFs. You may have to upgrade BIOS. This issue can also be resolved by adding the kernel parameter, pci=realloc. It enables reallocating PCI bridge resources to accommodate required resource. Edit the /etc/default/grub file and append the following kernel parameter to the existing GRUB_CMDLINE_LINUX line.
    pci=realloc
    Update grub and reboot the server.
    grub2-mkconfig -o /boot/grub2/grub.cfg
    reboot
    Verify the kernel parameters.
    cat /proc/cmdline
  5. Step 2 may fail with the following error.
    echo $NUMVF > /sys/bus/pci/devices/$DEV/sriov_numvfs
    -bash: /sys/bus/pci/devices/<device_ID>/sriov_numvfs: cannot overwrite existing file
    It is most likely because the noclobber option is enabled by default. To check this option, run:
    set -o | grep noclobber
    An output example
    noclobber       on
    This means that overwriting the contents of a file using shell redirection ( the > and >> operators) is not allowed in a command line interface. To disable the noclobber option, run:
    set +o noclobber
    Or you can bypass the noclobber restriction by using the >| redirection operator.
    echo $NUMVF >| /sys/bus/pci/devices/$DEV/sriov_numvfs
  6. Verify the Virtual Functions with the command lspci | grep Napatech.
    This is an example of the lspci | grep Napatech.
    # lspci | grep Napatech
    3b:00.0 Network controller: Napatech A/S NT200A02 Network Adapter
    
    # echo 3 > /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs
    
    # lspci | grep Napatech
    3b:00.0 Network controller: Napatech A/S NT200A02 Network Adapter
    3b:00.4 Network controller: Napatech A/S Device 051a
    3b:00.5 Network controller: Napatech A/S Device 051a
    3b:00.6 Network controller: Napatech A/S Device 051a
    3b:00.7 Network controller: Napatech A/S Device 051a
    3b:01.0 Network controller: Napatech A/S Device 051a
    Note: The Virtual Functions will be bound to devices XX:XX.4 and up.
  7. Bind all Virtual Functions to the VFIO driver.
    • Enter the following command for an NT200A02 SmartNIC:
      echo 18f4 051a > /sys/bus/pci/drivers/vfio-pci/new_id
    • Enter the following command for an NT50B01 SmartNIC:
      echo 18f4 051b > /sys/bus/pci/drivers/vfio-pci/new_id
  8. Verify the kernel driver using the command lspci -k | grep -A1 Napatech.
    # lspci -k | grep -A1 "Napatech"
    3b:00.0 Network controller: Napatech A/S Device 05a1
    	Subsystem: Napatech A/S Device 0001
    	Kernel driver in use: nt3gd
    3b:00.4 Network controller: Napatech A/S Device 051a
    	Subsystem: Napatech A/S Device 0001
    	Kernel driver in use: vfio-pci
    3b:00.5 Network controller: Napatech A/S Device 051a
    	Subsystem: Napatech A/S Device 0001
    	Kernel driver in use: vfio-pci
    3b:00.6 Network controller: Napatech A/S Device 051a
    	Subsystem: Napatech A/S Device 0001
    	Kernel driver in use: vfio-pci