Deduplication

Feature Set N-ANL11

Platform
Intel® PAC
Napatech SmartNIC
Content Type
Feature Description

In this chapter

This chapter describes the deduplication functionality, which enables discarding or retransmission of duplicate or nonduplicate frames.

Note: This chapter does not apply to NT200A01 running at 2 × 100 Gbit/s nor 2 × 25 Gbit/s, to NT100E3-1-PTP, to NT40E3-4-PTP running on the capture/replay image nor to NT40A01 running on the 4 × 10/1 Gbit/s SLB image.

Implementation principle

The deduplication feature is based on a filter setup which can be used to select which part of the incoming traffic is to be subjected to deduplication, such as traffic on a specified port or of a specified type (see Filtering).

Duplicate frames are usually not 100% identical. They have typically passed through the network along different routes and therefore have different MAC source and destination addresses and possibly also different MPLS labels. Therefore deduplication can be set up with different offsets so that only a part of a frame is compared to other frames to see if the frame parts in question are identical.

The comparison of frames to determine which frames are duplicates of previously received frames is done using a correlation key recipe, which means that all protocol offsets and masking settings available here can be used to determine which parts of the frames are compared (see Correlation Key). The scope for the deduplication is further specified by assigning a group ID to frames compared by the recipe.

It can be configured whether duplicate, nonduplicate or no frames are to be dropped and/or retransmitted. Furthermore a color can be assigned to all duplicate frames. These actions will be performed only on frames with the specified group ID.

Deduplication only takes place within a specified time window.

Recognition of duplicate frames

A frame is recognized as a duplicate frame by the SmartNIC, if all of these conditions are fulfilled:

  • The frames must belong to the same specified part of the traffic.
  • The whole frame or a specified part of it must have the same correlation key as another frame or frame part (see Correlation Key).
  • The two frames must have the same group ID.
  • The time between the reception of the two frames must not exceed the specified deduplication window. This window can vary from 10 μs to 2 s.

Deduplication configuration

Deduplication is configured using NTPL (Napatech programming language – see DN-0449), except for the deduplication window, which is specified by the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449).

Deduplication example

These NTPL expressions will set up deduplication based on the Layer 3 payload of an IP packet:
DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Payload == IP

Local retransmission example

These NTPL expressions will cause duplicate frames received on port 0 to be retransmitted on port 1:
DeduplicationConfig[retransmit=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Deduplication example reusing a group ID

A group ID can be reused for more than one filter. This is an advantage, for instance, in a setup where the same Ethernet frames are received both in raw form and tunneled. These NTPL expressions will calculate a correlation key on the full frame content and one on the inner content and drop all duplicates:
DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Deduplication example using multiple group IDs

Multiple group IDs can be used for scenarios requiring individual deduplication on, for instance, multiple ports. This allows otherwise identical packets to be received on different ports without being recognized as duplicates. These NTPL expressions will separate the deduplication on the L3 payload between ports 1 and 2:
DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1

Deduplication statistics

The deduplication functionality generates per-port statistics over the number of frames being discarded, retransmitted and detected as duplicates.

Guidelines on the deduplication window

The maximum value that the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449) should be set to varies depending on the SmartNIC, the average incoming frame size and the traffic load.

This table shows the maximum values for different SmartNICs with different images, an average frame size of 300 bytes and 100% traffic load (line rate multiplied by the number of ports). As a rule of thumb the maximum window size is inversely proportional to the load percentage and directly proportional to the average frame size. So, for a load of 50%, or for an average frame size of 600 bytes, the values in the table can be multiplied by 2.

SmartNIC FPGA Image / AFU Maximum Window Size [in μs]
NT200A02 2 × 100 Gbit/s 100
NT200A02 2 × 40 Gbit/s 200
NT200A02 8 × 10 Gbit/s 200
NT200A01 2 × 40 Gbit/s 90
NT200A01 8 × 10 Gbit/s 90
NT80E3-2-PTP 2 × 40 Gbit/s 90
NT80E3-2-PTP 8 × 10 Gbit/s 90
Intel® PAC A10 GX with Napatech Link™ Capture Software 1 × 40 Gbit/s 100
Intel® PAC A10 GX with Napatech Link™ Capture Software 4 × 10 Gbit/s 100
NT40E3-4-PTP 4 × 10 Gbit/s 100
NT20E3-2-PTP 2 × 10 Gbit/s 800
NT40A01-4×1 4 × 1 Gbit/s 1600