Use this procedure to run the forward application with the custom main-app container in the Kubernetes environment.
Before you begin
- Deployed the Napatech device-plugin pod. See Deploying the Napatech Device-Plugin Pod.
- Built the container images for applications. See Building Demo-Application Container Images.
- The traffic containing required VLAN IDs for the test. By default, the first virtual function is associated to VLAN ID 44, and the VLAN ID increments for subsequent virtual functions. It is configurable in the ntdevplugin.yaml file.
About this task
It is possible to run the forward application with the custom main-app container as shown in the following figure.
The ntmain application is used as a custom main application. In this test, the ntmain application is deployed in a stand-alone container/pod, which is connected to the Napatech DPDK PMD (ntnic), while the forward application instances in user-application containers/pods use the standard DPDK virtio PMD with the Napatech patch. In the ntmain application, the VLAN ID of each virtual function is passed to the Napatech DPDK PMD, which creates a table of VLAN IDs with associated virtual functions. After the forward application configures default flows, the SmartNIC starts to deliver frames to each virtual function. As shown in the figure, received frames with VLAN ID 44 and 45 are delivered to the first application pod via virtual function 4 and 5. Received frames with VLAN ID 46 and 47 are delivered to the second application pod via virtual function 6 and 7. After the forward application configures new flows for offloading to the SmartNIC, received frames on one port that match the learned flows are transmitted on another port without being delivered to the host. See Configuring flows in a user-application container.
Procedure
-
Delete any application containers, which are currently running.
For example:
kubectl delete -f kubernetes_v22.11.1/manifest/k8s-inline-upf-demo.yaml
An output example:deployment.apps "inline-upf-demo" deleted
-
Edit the ntdevplugin.yaml configuration file.
Set the MainApp parameter to "disabled" if it is currently enabled. Disabling the Napatech main-app container allows the custom main-app container to request the nt_pf resource to the Napatech device plugin.
cd /opt/ntinl/ vim kubernetes_v<version>/ntdevplugin/ntdevplugin.yaml
where version is the version identifier of the corresponding package.MainApp: "disabled"
-
Adjust other parameters if necessary.
apiVersion: v1 kind: ConfigMap metadata: name: napatech-device-plugin-config namespace: kube-system data: NicPci: "autodetect" NicAutoInit: "enabled" MainApp: "disabled" vfCount: "4" vfMacStart: "02:11:22:33:44:00" vfVlanStart: "44" vfNumberOfQueues: "1" immutable: true
where:- vfCount: "4": Specifies the number of virtual functions to be enabled.
- vfMacStart: "02:11:22:33:44:00": Specifies the MAC address of the first virtual function.
- vfVlanStart: "44": Specifies the VLAN ID of the first virtual function. By default, the first virtual function is associated to VLAN ID 44, and the VLAN ID increments for subsequent virtual functions.
- vfNumberOfQueues: "1": Specifies the number of queues per virtual function. By default, one queue is enabled per virtual function.
For more information on the supported parameters in the ntdevplugin.yaml file, see ConfigMap.
-
Delete the Napatech device-plugin container, after making changes in the
ntdevplugin.yaml file.
Skip this step if the ntdevplugin.yaml file has not been changed.
Whenever the ntdevplugin.yaml file is changed, the Napatech device-plugin container must be deleted and restarted.kubectl delete -f kubernetes_v<version>/ntdevplugin/ntdevplugin.yaml
where version is the version identifier of the corresponding package. For example:kubectl delete -f kubernetes_v22.11.1/ntdevplugin/ntdevplugin.yaml
An output example:configmap "napatech-device-plugin-config" deleted daemonset.apps "napatech-device-plugin" deleted
Deploy the Napatech device-plugin container. For example:kubectl apply -f kubernetes_v22.11.1/ntdevplugin/ntdevplugin.yaml
An output example:configmap/napatech-device-plugin-config created daemonset.apps/napatech-device-plugin created
Reboot the system.reboot
-
Deploy the custom main-app container using the
k8s-custom-main.yaml file.
For example:
kubectl apply -f kubernetes_v22.11.1/manifest/k8s-custom-main.yaml
An output example:deployment.apps/k8s-custom-main created
-
Deploy the forward container if it is not deployed.
For example:
kubectl apply -f kubernetes_v22.11.1/manifest/k8s-forward.yaml
An output example:deployment.apps/forward created
-
Check whether the deployed pods are running.
kubectl get pods -A
This command retrieves a list of pods from all namespaces in a Kubernetes cluster. An output example:NAMESPACE NAME READY STATUS RESTARTS AGE kube-system helm-install-traefik-79rrr 0/1 Completed 1 56d kube-system helm-install-traefik-crd-xtdj9 0/1 Completed 0 56d kube-system svclb-traefik-5fecb96c-whwnn 2/2 Running 12 (13m ago) 56d kube-system napatech-device-plugin-rkw6q 2/2 Running 2 (13m ago) 20m kube-system coredns-77ccd57875-4kxvk 1/1 Running 6 (13m ago) 56d kube-system traefik-64f55bb67d-wlq42 1/1 Running 6 (13m ago) 56d kube-system local-path-provisioner-957fdf8bc-xgrkk 1/1 Running 7 (12m ago) 56d kube-system metrics-server-648b5df564-2l7lp 0/1 Running 7 (12m ago) 56d default custom-main-77d45c89cb-c7ldd 1/1 Running 0 10m default forward-8b4cfdd97-srbbk 1/1 Running 0 5m29s default forward-8b4cfdd97-dmw8s 1/1 Running 0 5m29s
The output shows that the Napatech device-plugin pod, the custom-main pod and two forward pods are running. Applications must be also running as the application pods are running. -
Check processes on the host.
For example:
ps ax | grep dpdk
An output example:5508 ? Sl 11:18 dpdk-forward -c 0xfff0 -n 4 --file-prefix=nt-container6 --no-pci --vdev=virtio_user0,path=/usr/local/var/run/stdvio6/stdvio6,server=1,mac=02:11:22:33:44:02,packed_vq=1,queues=1 --vdev=virtio_user1,path=/usr/local/var/run/stdvio7/stdvio7,server=1,mac=02:11:22:33:44:03,packed_vq=1,queues=1 -- -i 0 -e 1 -q 1 5509 ? Sl 11:18 dpdk-forward -c 0xfff0 -n 4 --file-prefix=nt-container4 --no-pci --vdev=virtio_user0,path=/usr/local/var/run/stdvio4/stdvio4,server=1,mac=02:11:22:33:44:00,packed_vq=1,queues=1 --vdev=virtio_user1,path=/usr/local/var/run/stdvio5/stdvio5,server=1,mac=02:11:22:33:44:01,packed_vq=1,queues=1 -- -i 0 -e 1 -q 1 5903 pts/0 S+ 0:00 grep --color=auto dpdk
The output shows that the ntmain application and two instances of the forward application are running. -
Check the log messages of the custom main-app pod to verify whether the
ntmain application is running properly.
kubectl logs -n <pod_namespace> <pod_full_name>
For example:kubectl logs -n default custom-main-77d45c89cb-c7ldd
This command retrieves the logs from the custom-main pod. An output example:POD Configuration from Napatech Device Plugin for resource 'napatech.com/nt_pf: 1' NT_PF_NUM = 0 NT_PF_DEV = 0000:42:00.0 VF_TOKEN = a6d57050-b500-4621-af8b-a8523e76bfa0 NT_VF_NUM = 4 5 6 7 NT_VF_QUEUES = 1 NT_VF_DEV = 0000:42:00.4 0000:42:00.5 0000:42:00.6 0000:42:00.7 NT_VF_MAC = 02:11:22:33:44:00 02:11:22:33:44:01 02:11:22:33:44:02 02:11:22:33:44:03 NT_VF_VLAN = 44 45 46 47 NT_VF_SOCKET = /usr/local/var/run/stdvio4/stdvio4 /usr/local/var/run/stdvio5/stdvio5 /usr/local/var/run/stdvio6/stdvio6 /usr/local/var/run/stdvio7/stdvio7 NT_DPDK_PREFIX = nt-container0 Executing: ntmain -c 0xfffe -n 4 --file-prefix=nt-container0 --vfio-vf-token=a6d57050-b500-4621-af8b-a8523e76bfa0 -a 0000:42:00.0,exception_path=1,portqueues=[4:1,5:1,6:1,7:1] -a 0000:42:00.4,vlan=44,sep=1 -a 0000:42:00.5,vlan=45,sep=1 -a 0000:42:00.6,vlan=46,sep=1 -a 0000:42:00.7,vlan=47,sep=1 -- --queues 1 Wait for ntmain with PID 8 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/nt-container0/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_ntnic (18f4:1c5) device: 0000:42:00.0 (socket 1) ETHDEV: WRN: iova mode (2) should be PA for performance reasons ETHDEV: INF: NT VFIO device setup 0000:42:00.0 NTHW: INF: PCI:0000:42:00.0: FPGA 0200-9563-55-16 (C8255B3710) [644B901E] … … VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) set queue enable: 1 to qp idx: 1 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) read message VHOST_USER_SET_STATUS VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) new device status(0x0000000f): VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -RESET: 0 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -ACKNOWLEDGE: 1 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -DRIVER: 1 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -FEATURES_OK: 1 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -DRIVER_OK: 1 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -DEVICE_NEED_RESET: 0 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) -FAILED: 0 VHOST_CONFIG: (/usr/local/var/run/stdvio7/stdvio7) virtio is now ready for processing. VDPA: INF: New port /usr/local/var/run/stdvio7/stdvio7, vDPA dev: 0000:42:00.7 VDPA: INF: Virtio Negotiated features 0000000d40008000 VDPA: INF: Running virtio in vDPA mode : "Packed-Ring" "In-Order" VDPA: INF: ntvf_vdpa_start: Number of VRINGs=2 VDPA: INF: DMA map, region 0: HVA 0x7f7640000000, GPA 0x140000000, size 0x40000000. VDPA: INF: ntvf_vdpa_create_vring: idx=0: vq.desc 0x7f767ff90000 VDPA: INF: ntvf_vdpa_create_vring: idx=1: vq.desc 0x7f767ff8b000
-
Transmit traffic on the traffic generator.
The traffic must contain required VLAN IDs.
-
Start the monitoring tool to view statistics..
kubectl -n kube-system exec \ -it daemonsets/napatech-device-plugin -c napatech-main-app -- monitoring
An output example:┌─────────────────────────── monitoring pci:0000:42:00.0 fpga:200-9563-55-16 ──────────────────────────┐ │ Port Type PCI Intf NIM Link Speed MTU Rx Mbps Tx Mbps Rx Mpps Tx Mpps │ │ * 0 phys 00.0 phy0 present up 100G 1500 50.00 49.99 0.03 0.03 ♦│ │ 1 phys 00.0 phy1 present up 100G 1500 49.99 49.99 0.03 0.03 ▒│ │ 2 virt 00.4 VF4 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 3 virt 00.5 VF5 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 4 virt 00.6 VF6 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 5 virt 00.7 VF7 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ MAC: 00:0d:e9:07:9d:17 ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Rx counters │ │ Packets : 00000000113399429 Octets : 00000016102718908 │ │ Broadcast : 00000000000000000 Multicast : 00000000000000000 │ │ Unicast : 00000000113399429 Crc errors : 00000000000000000 │ │ 64 octets : 00000000000000000 65-127 octets : 00000000000000000 │ │ 128-255 octets : 00000000113399429 256-511 octets : 00000000000000000 │ │ 512-1023 octets : 00000000000000000 1024-1518 octets: 00000000000000000 │ │ 1519-2047 octets: 00000000000000000 2048-4095 octets: 00000000000000000 │ │ 4096-8191 octets: 00000000000000000 8192-max octets : 00000000000000000 │ │ Undersize : 00000000000000000 Oversize : 00000000000000000 │ │ Fragments : 00000000000000000 Drop events : 00000000000000000 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ Reset Port TX/Rx Queue Color Flowstat
Press the q key on the keyboard to view statistics on each virtual function. An output example:┌─────────────────────────── monitoring pci:0000:42:00.0 fpga:200-9563-55-16 ──────────────────────────┐ │ Queue Owner Dest Packets Octets VQ Drop pkts VQ Drop octets │ │ * 0 phy0 ctl/in #000000000000000 #00000000000000000 #000000000000000 #00000000000000000 ♦│ │ 1 phy1 ctl/in #000000000000000 #00000000000000000 #000000000000000 #00000000000000000 ▒│ │ 2 VF4 ctl/in #000000027540934 #00000003910813208 #000000027409861 #00000003892200852 ▒│ │ 3 VF5 ctl/in #000000024750119 #00000003960019030 #000000024619047 #00000003939047510 ▒│ │ 4 VF6 ctl/in #000000000987809 #00000000130390788 #000000000985249 #00000000130052868 ▒│ │ 5 VF7 ctl/in #000000000381344 #00000000057201600 #000000000380128 #00000000057019200 ▒│ │ 6 -unused- #000000000000000 #00000000000000000 #000000000000000 #00000000000000000 ▒│ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ QueueId: 0 │ │ flush packets : 000000177055481 flush octets : 00000026654364008 │ │ drop packets : 000000000000000 drop octets : 00000000000000000 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ Reset Port TX/Rx Queue Color Flowstat
-
To delete the application pods, run:
kubectl delete -f kubernetes_v22.11.1/manifest/k8s-forward.yaml kubectl delete -f kubernetes_v22.11.1/manifest/k8s-custom-main.yaml
An output example:deployment.apps "forward" deleted deployment.apps "custom-main" deleted