Deduplication

In this chapter

This chapter describes the deduplication functionality, which enables discarding or retransmission of duplicate or nonduplicate frames.

Implementation principle

The deduplication feature is based on a filter setup which can be used to select which part of the incoming traffic is to be subjected to deduplication, such as traffic on a specified port or of a specified type (see Filtering).

Duplicate frames are usually not 100% identical. They have typically passed through the network along different routes and therefore have different MAC source and destination addresses and possibly also different MPLS labels. Therefore deduplication can be set up with different offsets so that only a part of a frame is compared to other frames to see if the frame parts in question are identical.

The comparison of frames to determine which frames are duplicates of previously received frames is done using a correlation key recipe, which means that all protocol offsets and masking settings available here can be used to determine which parts of the frames are compared (see Correlation Key). The scope for the deduplication is further specified by assigning a group ID to frames compared by the recipe.

It can be configured whether duplicate, nonduplicate or no frames are to be dropped and/or retransmitted. Furthermore a color can be assigned to all duplicate frames. These actions will be performed only on frames with the specified group ID.

Deduplication only takes place within a specified time window.

Recognition of duplicate frames

A frame is recognized as a duplicate frame by the SmartNIC, if all of these conditions are fulfilled:

• The frames must belong to the same specified part of the traffic.
• The whole frame or a specified part of it must have the same correlation key as another frame or frame part (see Correlation Key).
• The two frames must have the same group ID.
• The time between the reception of the two frames must not exceed the specified deduplication window. This window can vary from 10 μs to 2 s.

Deduplication configuration

Deduplication is configured using NTPL (Napatech programming language – see DN-0449), except for the deduplication window, which is specified by the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449).

Deduplication example

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Payload == IP

Local retransmission example

DeduplicationConfig[retransmit=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Deduplication example reusing a group ID

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Deduplication example using multiple group IDs

DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1

Deduplication statistics

The deduplication functionality generates per-port statistics over the number of frames being discarded, retransmitted and detected as duplicates.

Guidelines on the deduplication window

The maximum value that the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449) should be set to varies depending on the SmartNIC, the average incoming frame size and the traffic load.

This table shows the maximum values for different SmartNICs with different images, an average frame size of 300 bytes and 100% traffic load (line rate multiplied by the number of ports). As a rule of thumb the maximum window size is inversely proportional to the load percentage and directly proportional to the average frame size. So, for a load of 50%, or for an average frame size of 600 bytes, the values in the table can be multiplied by 2.