Use this procedure to run the forward application with the Napatech main-app Container in the Kubernetes environment.
Before you begin
- Deployed the Napatech device-plugin pod. See Deploying the Napatech Device-Plugin Pod.
- Built the container images for applications. See Building Demo-Application Container Images.
- Traffic containing required VLAN IDs for the test. By default, the first virtual function is associated to VLAN ID 44, and the VLAN ID increments for subsequent virtual functions. It is configurable in the ntdevplugin.yaml file.
About this task
By default, the forward application runs with the Napatech main-app container as illustrated in the following figure.
The ntmain application in the Napatech main-app container is attached to the Napatech DPDK PMD (ntnic), while the forward application instances in user-application pods use the standard DPDK virtio PMD with the Napatech patch. In the ntmain application, the VLAN ID of each virtual function is passed to the Napatech DPDK PMD which creates a table of VLAN IDs with associated virtual functions. After the forward application configures default flows, the SmartNIC starts to deliver frames to each virtual function. As shown in the figure, received frames with VLAN ID 44 and 45 are delivered to the first user application pod via virtual function 4 and 5. Received frames with VLAN ID 46 and 47 are delivered to the second user application pod via virtual function 6 and 7. After the forward application configures new flow rules for offloading to the SmartNIC, received frames on one port that match the learned flows are transmitted on another port without being delivered to the host. See Configuring flows in a user-application container.
Procedure
-
Delete any application containers, which are currently running.
For example:
kubectl delete -f kubernetes_v22.11.1/manifest/k8s-inline-test.yaml kubectl delete -f kubernetes_v22.11.1/manifest/k8s-inline-upf-demo.yaml
An output example:deployment.apps "inline-test" deleted deployment.apps "inline-upf-demo" deleted
-
Edit the ntdevplugin.yaml configuration file.
vim kubernetes_v22.11.1/ntdevplugin/ntdevplugin.yaml
Set MainApp to "enabled" if it is disabled. It is enabled by default. Check the configuration file.MainApp: "enabled"
-
Adjust other parameters if necessary.
apiVersion: v1 kind: ConfigMap metadata: name: napatech-device-plugin-config namespace: kube-system data: NicPci: "autodetect" NicAutoInit: "enabled" MainApp: "enabled" vfCount: "4" vfMacStart: "02:11:22:33:44:00" vfVlanStart: "44" vfNumberOfQueues: "1" immutable: true
where:- vfCount: "4": Specifies the number of virtual functions to be enabled.
- vfMacStart: "02:11:22:33:44:00": Specifies the MAC address of the first virtual function.
- vfVlanStart: "44": Specifies the VLAN ID of the first virtual function. By default, the first virtual function is associated to VLAN ID 44, and the VLAN ID increments for subsequent virtual functions.
- vfNumberOfQueues: "1": Specifies the number of queues per virtual function. By default, one queue is enabled per virtual function.
For more information on the supported parameters in the ntdevplugin.yaml file, see ConfigMap.
-
Delete the Napatech device-plugin container, after making changes in the
ntdevplugin.yaml file.
Skip this step if the ntdevplugin.yaml file has not been changed.
Whenever the ntdevplugin.yaml file is changed, the Napatech device-plugin container must be deleted and restarted. For example:kubectl delete -f kubernetes_v<version>/ntdevplugin/ntdevplugin.yaml
An output example:configmap "napatech-device-plugin-config" deleted daemonset.apps "napatech-device-plugin" deleted
Deploy the Napatech device-plugin container. For example:kubectl apply -f kubernetes_v22.11.1/ntdevplugin/ntdevplugin.yaml
An output example:configmap/napatech-device-plugin-config created daemonset.apps/napatech-device-plugin created
Reboot the system.reboot
-
Apply the forward containers.
For example:
kubectl apply -f kubernetes_v22.11.1/manifest/k8s-forward.yaml
An output example:deployment.apps/forward created
This will start two instances of the forward application. -
Check whether the application pods are running.
kubectl get pods -A
This command retrieves a list of pods from all namespaces in a Kubernetes cluster. An output example:NAMESPACE NAME READY STATUS RESTARTS AGE kube-system helm-install-traefik-79rrr 0/1 Completed 1 56d kube-system helm-install-traefik-crd-xtdj9 0/1 Completed 0 56d kube-system svclb-traefik-5fecb96c-whwnn 2/2 Running 10 (19d ago) 56d kube-system local-path-provisioner-957fdf8bc-xgrkk 1/1 Running 5 (19d ago) 56d kube-system metrics-server-648b5df564-2l7lp 0/1 Running 5 (19d ago) 56d kube-system coredns-77ccd57875-4kxvk 1/1 Running 5 (19d ago) 56d kube-system traefik-64f55bb67d-wlq42 1/1 Running 5 (19d ago) 56d kube-system napatech-device-plugin-pts2d 2/2 Running 0 17h default forward-8b4cfdd97-r4m7m 1/1 Running 0 22s default forward-8b4cfdd97-9hc5g 1/1 Running 0 22s
The output shows that two forward pods are running -
Check the log messages of pods to verify whether NT SmartNIC was properly detected and
both plugin and ntmain containers have started correctly.
kubectl logs -n <pod_namespace> <pod_full_name>
For example:kubectl logs -n kube-system napatech-device-plugin-pts2d
An output example:Defaulted container "napatech-device-plugin" out of: napatech-device-plugin, napatech-main-app, napatech-init (init) I0925 15:01:32.307864 1 ntdevplugin.go:667] Napatech Device Plugin version 1.5 I0925 15:01:32.308101 1 ntdevplugin.go:669] Configuration: I0925 15:01:32.308121 1 ntdevplugin.go:671] NicPci : autodetect I0925 15:01:32.308169 1 ntdevplugin.go:671] vfMacStart : 02:11:22:33:44:00 I0925 15:01:32.308195 1 ntdevplugin.go:671] vfVlanStart : 44 I0925 15:01:32.308215 1 ntdevplugin.go:671] vfNumberOfQueues : 1 I0925 15:01:32.308235 1 ntdevplugin.go:671] MainApp : enabled I0925 15:01:32.522022 1 ntdevplugin.go:687] NT SmartNIC detected at PCI address: 0000:42:00.0 I0925 15:01:32.716362 1 ntdevplugin.go:620] Found: PF: [0000:42:00.0] I0925 15:01:32.716451 1 ntdevplugin.go:621] VFs: [0000:42:00.4 0000:42:00.5 0000:42:00.6 0000:42:00.7] I0925 15:01:32.716477 1 ntdevplugin.go:733] ntPhysFuncManager is disabled by configuration I0925 15:01:32.716507 1 ntdevplugin.go:302] Creating VF device: 0000:42:00.7 I0925 15:01:32.716520 1 ntdevplugin.go:302] Creating VF device: 0000:42:00.4 I0925 15:01:32.716531 1 ntdevplugin.go:302] Creating VF device: 0000:42:00.5 I0925 15:01:32.716556 1 ntdevplugin.go:302] Creating VF device: 0000:42:00.6 … …
To retrieve the log messages of the napatech-init container, run:kubectl logs -n kube-system napatech-device-plugin-pts2d -c napatech-init
An output example:NT SmartNIC info: NIC PCI address: 0000:42:00.0 NIC type: Network controller: Napatech A/S NT200A02 Network Adapter NIC driver: vfio-pci VFs requested: 4 VFs configured: 4 NT SmartNIC 0000:42:00.0 is configured.
To retrieve the log messages of the napatech-main-app container, run:kubectl logs -n kube-system napatech-device-plugin-pts2d -c napatech-main-app
An output example:Execution of NT main APP: NT_NIC = 0000:42:00.0 NT_VF_DEV = 0000:42:00.4 0000:42:00.5 0000:42:00.6 0000:42:00.7 NT_DPDK_PREFIX = nt-main-0000:42:00.0 NT_VF_NUM = 4 NT_VF_QUEUES = 1 VF_VLAN_START = 44 VF_TOKEN = 711081f2-1054-46b2-94b9-b53a10c979e1 Executing: ntmain -c 0xfffe -n 4 --file-prefix=nt-main-0000:42:00.0 --vfio-vf-token=711081f2-1054-46b2-94b9-b53a10c979e1 -a 0000:42:00.0,exception_path=1,portqueues=[4:1,5:1,6:1,7:1] -a 0000:42:00.4,vlan=44,sep=1 -a 0000:42:00.5,vlan=45,sep=1 -a 0000:42:00.6,vlan=46,sep=1 -a 0000:42:00.7,vlan=47,sep=1 -- --queues 1 Wait for ntmain with PID 20 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/nt-main-0000:42:00.0/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_ntnic (18f4:1c5) device: 0000:42:00.0 (socket 1) … …
-
Check processes on the host.
ps ax | grep dpdk
An output example:1132256 ? Sl 29:10 dpdk-forward -c 0xfff0 -n 4 --file-prefix=nt-container4 --no-pci --vdev=virtio_user0,path=/usr/local/var/run/stdvio4/stdvio4,server=1,mac=02:11:22:33:44:00,packed_vq=1,queues=1 --vdev=virtio_user1,path=/usr/local/var/run/stdvio5/stdvio5,server=1,mac=02:11:22:33:44:01,packed_vq=1,queues=1 -- -i 0 -e 1 -q 1 1132257 ? Sl 29:10 dpdk-forward -c 0xfff0 -n 4 --file-prefix=nt-container6 --no-pci --vdev=virtio_user0,path=/usr/local/var/run/stdvio6/stdvio6,server=1,mac=02:11:22:33:44:02,packed_vq=1,queues=1 --vdev=virtio_user1,path=/usr/local/var/run/stdvio7/stdvio7,server=1,mac=02:11:22:33:44:03,packed_vq=1,queues=1 -- -i 0 -e 1 -q 1 1133151 pts/1 S+ 0:00 grep --color=auto dpdk
By default, two instances of the forward application are executed. The output shows that they are up and running. -
Check log messages of the forward pods to verify whether two instances of the
forward application are running properly.
For example:
kubectl logs -n default forward-8b4cfdd97-r4m7m kubectl logs -n default forward-8b4cfdd97-9hc5g
An output example:POD Configuration from Napatech Device Plugin for resource 'napatech.com/nt_vf: 2' NT_VF_NUM = 4 5 NT_VF_QUEUES = 1 NT_VF_DEV = 0000:42:00.4 0000:42:00.5 NT_VF_MAC = 02:11:22:33:44:00 02:11:22:33:44:01 NT_VF_VLAN = 44 45 NT_VF_SOCKET = /usr/local/var/run/stdvio4/stdvio4 /usr/local/var/run/stdvio5/stdvio5 NT_DPDK_PREFIX = nt-container4 Executing: dpdk-forward -c 0xfff0 -n 4 --file-prefix=nt-container4 --no-pci --vdev=virtio_user0,path=/usr/local/var/run/stdvio4/stdvio4,server=1,mac=02:11:22:33:44:00,packed_vq=1,queues=1 --vdev=virtio_user1,path=/usr/local/var/run/stdvio5/stdvio5,server=1,mac=02:11:22:33:44:01,packed_vq=1,queues=1 -- -i 0 -e 1 -q 1 Wait for dpdk-forward with PID 10 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/nt-container4/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized vhost_user_start_server(): (/usr/local/var/run/stdvio4/stdvio4) waiting for client connection... vhost_user_start_server(): (/usr/local/var/run/stdvio5/stdvio5) waiting for client connection... Forward DPDK app - Forwarding packets between two ports. -------------------------------------------------------. Ingress port: 0 Eegress port: 1 Set filter: yes Number of queues: 1 HW offload: yes Decap packets: no Debug print: disabled Number of ports found: 2 Port 0 MAC: 02 11 22 33 44 00 Virtio path (0): /usr/local/var/run/stdvio4/stdvio4 Virtual port (0): 4 Server version 0.1 PCI ID: 0000:42:00.0 Meter version 0.1 Port 1 MAC: 02 11 22 33 44 01 Virtio path (1): /usr/local/var/run/stdvio5/stdvio5 Virtual port (1): 5 Core 5 RX port 0 -> TX port 1 using queue 0 Core 6 RX port 1 -> TX port 0 using queue 0 Core 7 acting as worker core for learn/unlearn Core 4 acting as worker core for statistics output Press U or send SIGNAL 10 to unlearn flows Press L or send SIGNAL 12 to learn flows Press 1 or send SIGNAL 40 to delete default flows Press 2 or send SIGNAL 41 to setup default flows Press Q or send SIGNAL 15 to quit Process ID: 10 Usage example: kill -10 10 Q: 1 R0.0: 0 F: 0 R1.0: 0 F: 0
-
Transmit traffic on the traffic generator.
The traffic must contain required VLAN IDs.
-
Start the monitoring tool to view statistics.
For example:
kubectl -n kube-system exec \ -it daemonsets/napatech-device-plugin -c napatech-main-app -- monitoring
An output example:┌─────────────────────────── monitoring pci:0000:42:00.0 fpga:200-9563-55-16 ──────────────────────────┐ │ Port Type PCI Intf NIM Link Speed MTU Rx Mbps Tx Mbps Rx Mpps Tx Mpps │ │ * 0 phys 00.0 phy0 present up 100G 1500 59.99 60.00 0.04 0.04 ♦│ │ 1 phys 00.0 phy1 present up 100G 1500 60.00 59.99 0.04 0.04 ▒│ │ 2 virt 00.4 VF4 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 3 virt 00.5 VF5 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 4 virt 00.6 VF6 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ 5 virt 00.7 VF7 packed up N/A 1500 0.00 0.00 0.00 0.00 ▒│ │ ▒│ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ MAC: 00:0d:e9:07:9d:17 ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Rx counters │ │ Packets : 00000000000283717 Octets : 00000000037450644 │ │ Broadcast : 00000000000000000 Multicast : 00000000000000000 │ │ Unicast : 00000000000283717 Crc errors : 00000000000000000 │ │ 64 octets : 00000000000000000 65-127 octets : 00000000000000000 │ │ 128-255 octets : 00000000000283717 256-511 octets : 00000000000000000 │ │ 512-1023 octets : 00000000000000000 1024-1518 octets: 00000000000000000 │ │ 1519-2047 octets: 00000000000000000 2048-4095 octets: 00000000000000000 │ │ 4096-8191 octets: 00000000000000000 8192-max octets : 00000000000000000 │ │ Undersize : 00000000000000000 Oversize : 00000000000000000 │ │ Fragments : 00000000000000000 Drop events : 00000000000000000 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ Reset Port TX/Rx Queue Color Flowstat
Press the q key on the keyboard to view statistics on each virtual function. An output example:┌─────────────────────────── monitoring pci:0000:42:00.0 fpga:200-9563-55-16 ──────────────────────────┐ │ Queue Owner Dest Packets Octets VQ Drop pkts VQ Drop octets │ │ * 0 phy0 ctl/in #000000000000001 #00000000000000132 #000000000000000 #00000000000000000 ♦│ │ 1 phy1 ctl/in #000000000000000 #00000000000000000 #000000000000000 #00000000000000000 ▒│ │ 2 VF4 ctl/in #000000000522060 #00000000074132380 #000000000512972 #00000000072841864 ▒│ │ 3 VF5 ctl/in #000000002424605 #00000000387936810 #000000002413981 #00000000386236970 ▒│ │ 4 VF6 ctl/in #000000004841192 #00000000639037344 #000000004831144 #00000000637711008 ▒│ │ 5 VF7 ctl/in #000000004440106 #00000000666015900 #000000004429866 #00000000664479900 ▒│ │ 6 -unused- #000000000000000 #00000000000000000 #000000000000000 #00000000000000000 ▒│ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ QueueId: 0 │ │ flush packets : 000000007221413 flush octets : 00000001067913302 │ │ drop packets : 000000000000000 drop octets : 00000000000000000 │ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ Reset Port TX/Rx Queue Color Flowstat
-
Run the following command to remove the forward containers.:
kubectl delete -f kubernetes_v<version>/manifest/k8s-forward.yaml
where version is the version identifier of the corresponding package. For example:kubectl delete -f kubernetes_v22.11.1/manifest/k8s-forward.yaml
An output example:deployment.apps "forward" deleted