In this chapter
This chapter describes the deduplication functionality, which enables discarding or retransmission of duplicate or nonduplicate frames.
The deduplication feature is based on a filter setup which can be used to select which part of the incoming traffic is to be subjected to deduplication, such as traffic on a specified port or of a specified type (see Filtering).
Duplicate frames are usually not 100% identical. They have typically passed through the network along different routes and therefore have different MAC source and destination addresses and possibly also different MPLS labels. Therefore deduplication can be set up with different offsets so that only a part of a frame is compared to other frames to see if the frame parts in question are identical.
The comparison of frames to determine which frames are duplicates of previously received frames is done using a correlation key recipe, which means that all protocol offsets and masking settings available here can be used to determine which parts of the frames are compared (see Correlation Key). The scope for the deduplication is further specified by assigning a group ID to frames compared by the recipe.
It can be configured whether duplicate, nonduplicate or no frames are to be dropped and/or retransmitted. Furthermore a color can be assigned to all duplicate frames. These actions will be performed only on frames with the specified group ID.
Deduplication only takes place within a specified time window.
Recognition of duplicate frames
A frame is recognized as a duplicate frame by the SmartNIC, if all of these conditions are fulfilled:
- The frames must belong to the same specified part of the traffic.
- The whole frame or a specified part of it must have the same correlation key as another frame or frame part (see Correlation Key).
- The two frames must have the same group ID.
- The time between the reception of the two frames must not exceed the specified deduplication window. This window can vary from 10 μs to 2 s.
Deduplication is configured using NTPL (Napatech programming language – see DN-0449), except for the deduplication window, which is specified by the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449).
DeduplicationConfig[drop=duplicate] = GroupID == 0 Define ckRecipe = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Payload == IP
Local retransmission example
DeduplicationConfig[retransmit=duplicate] = GroupID == 0 Define ckRecipe = CorrelationKey(Begin=StartOfFrame, End=EndOfFrame, DeduplicationGroupID=0) Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0
Deduplication example reusing a group ID
DeduplicationConfig[drop=duplicate] = GroupID == 0 Define ckRecipe1 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header, End=InnerLayer3PayloadEnd, DeduplicationGroupID=0) Assign[StreamID=0; CorrelationKey=ckRecipe1] = All Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0
Deduplication example using multiple group IDs
DeduplicationConfig[drop=duplicate] = GroupID == 0 DeduplicationConfig[drop=duplicate] = GroupID == 1 Define ckRecipe1 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=0) Define ckRecipe2 = CorrelationKey(Begin=Layer3Header, End=Layer3PayloadEnd, DeduplicationGroupID=1) Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0 Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1
The deduplication functionality generates per-port statistics over the number of frames being discarded, retransmitted and detected as duplicates.
Guidelines on the deduplication window
The deduplication window should be as small as possible for the use case in question. The maximum value that the DeduplicationWindow parameter in the ntservice.ini file (see ntservice.ini) should be set depending on the SmartNIC, the average incoming frame size and the traffic load.
See DeduplicationConfig command which includes the table with the maximum values for different SmartNICs with different FPGA images.