Deduplication

Napatech Link-Capture™ Software Features

Platform
Intel® PAC
Napatech SmartNIC
Content Type
Feature Description
Capture Software Version
Link™ Capture Software 12.10

In this chapter

This chapter describes the deduplication functionality, which enables discarding or retransmission of duplicate or nonduplicate frames.

Note: This chapter does not apply to the following SmartNICs.
  • NT200A02 running on the 2 × 100 Gbit/s FM1 image
  • NT200A02 running on the 2 × 40 Gbit/s FM1 image
  • NT200A02 running on the 8 × 10 Gbit/s FM1 image
  • NT100A01 running on the 4 × 25/10 Gbit/s FM1 image
  • NT100A01 running on the 4 × 10/1 Gbit/s FM1 image
  • NT40E3-4-PTP running on the capture/replay image
  • NT40A01 running on the 4 × 10/1 Gbit/s SLB image.

Implementation principle

The deduplication feature is based on a filter setup which can be used to select which part of the incoming traffic is to be subjected to deduplication, such as traffic on a specified port or of a specified type (see Filtering).

Duplicate frames are usually not 100% identical. They have typically passed through the network along different routes and therefore have different MAC source and destination addresses and possibly also different MPLS labels. Therefore deduplication can be set up with different offsets so that only a part of a frame is compared to other frames to see if the frame parts in question are identical.

The comparison of frames to determine which frames are duplicates of previously received frames is done using a correlation key recipe, which means that all protocol offsets and masking settings available here can be used to determine which parts of the frames are compared (see Correlation Key). The scope for the deduplication is further specified by assigning a group ID to frames compared by the recipe.

It can be configured whether duplicate, nonduplicate or no frames are to be dropped and/or retransmitted. Furthermore a color can be assigned to all duplicate frames. These actions will be performed only on frames with the specified group ID.

Deduplication only takes place within a specified time window.

Recognition of duplicate frames

A frame is recognized as a duplicate frame by the SmartNIC, if all of these conditions are fulfilled:

  • The frames must belong to the same specified part of the traffic.
  • The whole frame or a specified part of it must have the same correlation key as another frame or frame part (see Correlation Key).
  • The two frames must have the same group ID.
  • The time between the reception of the two frames must not exceed the specified deduplication window. This window can vary from 10 μs to 2 s.
Note: The calculations are made on a best-effort basis, and it might occur that a duplicate is not recognized, for instance, due to traffic load.

Detecting a duplicate

The probability of detecting a duplicate is depending on traffic load and the configured deduplication window (the maximum time between the reception of duplicates). The time stamp of a stored frame in the deduplication table is observed in the SmartNIC, and if the time exceeds the deduplication time of the window, the entry in the table is replaced with a new frame. If the deduplication window is set to a large value, duplicates of new received frames may not be recognized as it takes too long a time to expire existing frames in the table. If the deduplication window is set to a small value, a greater number of new frames can be detected as duplicates. On the other hand, reducing the deduplication window shortens the allowed time distance between duplicates. The period of the deduplication window is a trade-off between the number of new duplicates and the allowed time distance between duplicates. It is therefore important to tune the optimal deduplication window based on the network environment.

This graph is an example that shows the probability of detecting a duplicate based on the time between duplicates and the average frame length.


Page-1 Sheet.1

Maximum value on the deduplication window

The maximum value that the DeduplicationWindow parameter in the ntservice.ini file (see ntservice.ini) should be set depending on the SmartNIC, the average incoming frame size and the traffic load.

See the table with the maximum values of the deduplication window for different SmartNICs with different FPGA images at DeduplicationConfig command.

Deduplication configuration

Deduplication is configured using NTPL (Napatech programming language – see DN-0449), except for the deduplication window, which is specified by the DeduplicationWindow parameter in the ntservice.ini file (see DN-0449).

Deduplication example

These NTPL expressions will set up deduplication based on the Layer 3 payload of an IP packet:
DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Payload == IP

Local retransmission example

These NTPL expressions will cause duplicate frames received on port 0 to be retransmitted on port 1:
DeduplicationConfig[retransmit=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Deduplication example reusing a group ID

A group ID can be reused for more than one filter. This is an advantage, for instance, in a setup where the same Ethernet frames are received both in raw form and tunneled. These NTPL expressions will calculate a correlation key on the full frame content and one on the inner content and drop all duplicates:
DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Deduplication example using multiple group IDs

Multiple group IDs can be used for scenarios requiring individual deduplication on, for instance, multiple ports. This allows otherwise identical packets to be received on different ports without being recognized as duplicates. These NTPL expressions will separate the deduplication on the L3 payload between ports 1 and 2:
DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1

Deduplication statistics

The deduplication functionality generates per-port statistics over the number of frames being discarded, retransmitted and detected as duplicates.