In this chapter
This chapter describes the deduplication functionality, which enables discarding or retransmission of duplicate or nonduplicate frames.
- NT200A02 running on the 2 × 100 Gbit/s FM1 image
- NT200A02 running on the 2 × 40 Gbit/s FM1 image
- NT200A02 running on the 8 × 10 Gbit/s FM1 image
- NT100A01 running on the 4 × 25/10 Gbit/s FM1 image
- NT100A01 running on the 4 × 10/1 Gbit/s FM1 image
- NT40E3-4-PTP running on the capture/replay image
- NT40A01 running on the 4 × 10/1 Gbit/s SLB image.
The deduplication feature is based on a filter setup which can be used to select which part of the incoming traffic is to be subjected to deduplication, such as traffic on a specified port or of a specified type (see Filtering).
Duplicate frames are usually not 100% identical. They have typically passed through the network along different routes and therefore have different MAC source and destination addresses and possibly also different MPLS labels. Therefore deduplication can be set up with different offsets so that only a part of a frame is compared to other frames to see if the frame parts in question are identical.
The comparison of frames to determine which frames are duplicates of previously received frames is done using a correlation key recipe, which means that all protocol offsets and masking settings available here can be used to determine which parts of the frames are compared (see Correlation Key). The scope for the deduplication is further specified by assigning a group ID to frames compared by the recipe.
It can be configured whether duplicate, nonduplicate or no frames are to be dropped and/or retransmitted. Furthermore a color can be assigned to all duplicate frames. These actions will be performed only on frames with the specified group ID.
Deduplication only takes place within a specified time window.
Recognition of duplicate frames
A frame is recognized as a duplicate frame by the SmartNIC, if all of these conditions are fulfilled:
- The frames must belong to the same specified part of the traffic.
- The whole frame or a specified part of it must have the same correlation key as another frame or frame part (see Correlation Key).
- The two frames must have the same group ID.
- The time between the reception of the two frames must not exceed the specified deduplication window. This window can vary from 10 μs to 2 s.
Detecting a duplicate
The probability of detecting a duplicate is depending on traffic load and the configured deduplication window (the maximum time between the reception of duplicates). The time stamp of a stored frame in the deduplication table is observed in the SmartNIC, and if the time exceeds the deduplication time of the window, the entry in the table is replaced with a new frame. If the deduplication window is set to a large value, duplicates of new received frames may not be recognized as it takes too long a time to expire existing frames in the table. If the deduplication window is set to a small value, a greater number of new frames can be detected as duplicates. On the other hand, reducing the deduplication window shortens the allowed time distance between duplicates. The period of the deduplication window is a trade-off between the number of new duplicates and the allowed time distance between duplicates. It is therefore important to tune the optimal deduplication window based on the network environment.
This graph is an example that shows the probability of detecting a duplicate based on the time between duplicates and the average frame length.
Maximum value on the deduplication window
The maximum value that the DeduplicationWindow parameter in the ntservice.ini file (see ntservice.ini) should be set depending on the SmartNIC, the average incoming frame size and the traffic load.
See the table with the maximum values of the deduplication window for different SmartNICs with different FPGA images at DeduplicationConfig command.
Deduplication is configured using NTPL (Napatech programming language – see DN-0449), except for the deduplication window, which is specified by the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449).
DeduplicationConfig[drop=duplicate] = GroupID == 0 Define ckRecipe = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Payload == IP
Local retransmission example
DeduplicationConfig[retransmit=duplicate] = GroupID == 0 Define ckRecipe = CorrelationKey(Begin=StartOfFrame, End=EndOfFrame, DeduplicationGroupID=0) Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0
Deduplication example reusing a group ID
DeduplicationConfig[drop=duplicate] = GroupID == 0 Define ckRecipe1 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header, End=InnerLayer3PayloadEnd, DeduplicationGroupID=0) Assign[StreamID=0; CorrelationKey=ckRecipe1] = All Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0
Deduplication example using multiple group IDs
DeduplicationConfig[drop=duplicate] = GroupID == 0 DeduplicationConfig[drop=duplicate] = GroupID == 1 Define ckRecipe1 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Define ckRecipe2 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=1) Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0 Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1
The deduplication functionality generates per-port statistics over the number of frames being discarded, retransmitted and detected as duplicates.