User-Application Container

Default flows must be configured in the user application ensuring received frames are forwarded from the SmartNIC to each virtual function. For the code example how to configure default flows, see the forward.c file in the following directory of the Napatech Link-Inline™ Software package.

ntinl_package-<version>-linux/dpdk_v<version>/dpdk_v<version>-nt_<version>/examples/nt_examples/forward/

where version is the version identifier of the corresponding package.

The forward.c example illustrates how the IPv4 and IPv6 types are configured as default flows.

• Group 0 is used to filter IPv4 traffic using the RTE_FLOW_ITEM_TYPE_IPV4 pattern, and RTE_FLOW_ACTION_TYPE_JUMP is included to jump to group 1.
• Group 0 is used to filter IPv6 traffic using the RTE_FLOW_ITEM_TYPE_IPV6 pattern, and RTE_FLOW_ACTION_TYPE_JUMP is included to jump to group 2.

When IP traffic is delivered to the user application via each virtual function, group 1 is used for IPv4 flows and group 2 is used for IPv6 flows for offloading to the SmartNIC. See Flow configuration for offloading.

See also the following links:

• Group Attribute
• Correct sequence to define flow rules

For the code example how to configure flows for offloading, see the filter.c file in the following directory of the Napatech Link-Inline™ Software package.

ntinl_package-<version>-linux/dpdk_v<version>/dpdk_v<version>-nt_<version>/examples/nt_examples/forward/

where version is the version identifier of the corresponding package.

In a user application, DPDK EAL command-line parameters can be used to control various aspects of the user application's behavior, memory usage or core affinity as required. The following options are mandatory.

• --file-prefix: Specifies a prefix for all files created by DPDK, such as shared memory files, log files, and statistics files. This option is required to ensure uniqueness when multiple DPDK applications are running on the same system. It's helpful for identifying which DPDK application instance is generating the logs.
• --no-pci: Configures DPDK not to use any physical PCIe devices as the user application uses virtio interfaces.

In a user application, the --vdev option is used to select and start the virtio PMD.

--vdev=virtio_user<number>

where number represents the instance number of the virtio PMD. Each instance of the virtio PMD handles one virtual function. Consequently, if multiple virtual functions are connected to the user-application pod, multiple instances of the virtio PMD must run. The number for the first instance is 0, and it increments for the subsequent instances. For example:

--vdev=virtio_user0

The following parameters must be added to the --vdev option.

• path=<socket_if>: Specifies the vHost socket file directory.
  • If the shared vHost-user socket directory is used, set as follows:

    path=/usr/local/var/run/stdvio<VF_number>

    where VF_number is the virtual function. The VF_number for the first virtual function is 4, and it increments for the subsequent virtual functions. For example:

    path=/usr/local/var/run/stdvio4

  • If a separated directory is used, set as follows:

    /usr/local/var/run/stdvio<VF_number>/stdvio<VF_number>

    For example:

    path=/usr/local/var/run/stdvio4/stdvio5

  For information on the vHost-user socket directory options, see Shared/separated socket directory.
• server=1: The virtio PMD / vHost must run as a server.
• mac=<mac_address>: Specifies the MAC address for the virtual function. The MAC address of the first virtual function can be configured using the vfMacStart parameter in the ntdevplugin.yaml file.
• packed_vq=1: Only packed virtqueue is supported.
• queues=<number_of_queues>: Specifies the number of queues for RSS. It is set to 1 by default using the vfNumberOfQueues parameter in the ntdevplugin.yaml file.

These parameters are configured in the ConfigMap (the ntdevplugin.yaml file) and are passed to the user-application containers as environment variables. For more information on the supported parameters in the ntdevplugin.yaml file, see ConfigMap.

See the following links.

• Enabling RSS in the user application: API: Setting up Receive Side Scaling (RSS).
• Setting up multiple queues in the ntmain application: Setting up multiple queues.

The application pod must request resources to the Napatech device plugin by adding parameters to the manifest file. For example:

        resources:
          limits:
            napatech.com/nt_vf: 2
            hugepages-1Gi: 2Gi
            memory: 2Gi

See manifest file examples for demo application pods included in the following directory of the Napatech Link-Inline™ Software package.

ntinl_package-<version>-linux/kubernetes_v<version>/manifest/

where version is the version identifier of the corresponding package.

The Napatech device plugin mounts the following host directories on the user-application container. As the sep=1 option is used, each vHost-user socket file is placed in its individual directory. For information on the vHost-user socket directory options, see Shared/separated socket directory.

The Napatech device plugin also mounts the following host device files on the user-application container. where VF_group_number is the IOMMU group number for the vfio device of the assigned virtual functions.

The following parameters in the manifest file are supported to allocate the physical function and the number of virtual functions.

• napatech.com/nt_pf: 1: The physical function can be allocated to only one container. The target container will receive information about the physical function's PCI bus ID as well as details regarding all configured virtual functions.
• napatech.com/nt_vf: <number_of_vf>: number_of_vf specifies the number of virtual functions per container. The specified number of virtual functions can be allocated to multiple containers. If the requested number of virtual functions is not available, then the container will remain in a pending state. The valid value range is 1 to 128.

The following manifest snippet shows how to define an application container in a Kubernetes application pod.

…
…
    spec:
      containers:
      - name: forward
        image: k8s.io/ntforward:1.1
        imagePullPolicy: IfNotPresent
        command: ["/opt/kubernetes/start.sh"]
        resources:
          limits:
            napatech.com/nt_vf: 2
            hugepages-1Gi: 2Gi
            memory: 512Mi
        volumeMounts:
        - name: hugepage
          mountPath: /hugepages
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages

Using this manifest file, 2 virtual functions are requested for the container.

The provided manifest files of demo applications can be found in the following directory.

ntinl_package-<version>/kubernetes_v<version>/manifest/k8s-forward.yaml

where version is the version identifier of the corresponding package.

This script example can be used to start the forward application with the napatech.com/nt_vf: 2 request in the manifest file.

#!/usr/bin/env bash
NT_VF_MAC_ARRAY=($NT_VF_MAC)
NT_VF_SOCKET_ARRAY=($NT_VF_SOCKET)

# prepare dpdk vdev configuration for allocated virtual functions
APP_VDEV=""
for i in ${!NT_VF_MAC_ARRAY[@]}; do
        APP_VDEV="$APP_VDEV \
--vdev=virtio_user${i},path=${NT_VF_SOCKET_ARRAY[$i]},server=1,mac=${NT_VF_MAC_ARRAY[$i]},packed_vq=1,queues=${NT_VF_QUEUES}"
        # remove old socket
        rm -f ${NT_VF_SOCKET_ARRAY[$i]}
done

# Execute app
dpdk-forward -c 0xfff0 -n 4 --no-pci ${APP_VDEV} --file-prefix=${NT_DPDK_PREFIX} -- -i 0 -e 1 -q ${NT_VF_QUEUES}