DeduplicationConfig Command

Reference Documentation

Intel® PAC
Napatech SmartNIC
Content Type
Reference Information
Capture Software Version
Link™ Capture Software 12.10
Napatech Software Suite: DeduplicationConfig Command
DeduplicationConfig Command

The deduplication feature enables an adapter to selectively perform deduplication on traffic based on filter setup.

Internally, the feature uses Correlation Key Recipe to determine whether an incoming packet is a duplicate of a previously received one. Consequently, all protocol offset and masking settings available here can be used to set up the scope of deduplication.

The syntax for the DeduplicationConfig command is as follows:

<DeduplicationConfigAction>     ::= 'DeduplicationConfig' { '[' <DedulicationConfigOptionList> ']' } '=' <DeduplicationGroupTest>
<DeduplicationConfigOptionList> ::= <DeduplicationConfigOption> { ';' <DeduplicationConfigOption> }
<DeduplicationConfigOption>     ::= 'Drop'       '=' ('Duplicate' | 'Nonduplicate') |
                                    'Retransmit' '=' ('Duplicate' | 'Nonduplicate') |
                                    'Colorbit'   '=' ('0' | '1' | ... | '31')
<DeduplicationGroupTest>        ::= 'GroupID' '==' <DeduplicationGroupNumber>

'Drop' will determine which packets are dropped, and can be set to either Duplicate or Nonduplicate
'Retransmit' will determine which packets are retransmitted, and can be set to either Duplicate or Nonduplicate
'Colorbit' will set the provided colorbit on packets considered duplicates.

The 'GroupID' parameter constitutes a second criteria for packets to be considered a match. In other words, both the calulated CorrelationKey, and the GroupID has to match an earlier received packet. Thus, it is up to the specific application whether different or identical GroupIDs are to be used for different filters. A relevant case where identical Group IDs could be employed would be simultaneous matching on tunnelled and untunnelled traffic.

In addition to the settings configured by the DeduplicationConfig command, there is an ini-setting named 'DeduplicationWindow', which determines the expiration period for deduplication. Thus, two packets with matching CorrelationKey and GroupID will have to arrive within this time window in order for the second packet to be considered a duplicate.

Notes on DeduplicationWindow parameter

The precision of duplicate detection in our adapters depends on the product, the average incoming frame size, and the traffic load. The values in the table below represent the maximum recommended window size allowing a 99.9% duplicate detection, with 100% load on all ports and an average frame size of 300 bytes.

Adapter Line rate Window size at 100% load
NT200A02 2x100G120 µs
2x40G300 µs
8x10G300 µs
4x25G240 µs
2x25G480 µs
2x10G1200 µs
2x1G12000 µs
NT200A01 8x10G88 µs
2x40G88 µs
NT100A01 4x25G240 µs
4x10G600 µs
4x1G6000 µs
NT80E3-2-PTP 2x40G88 µs
8x10G88 µs
Intel PAC A10 GX 1x40G175 µs
4x10G175 µs
NT50B01 2x25G480 µs
2x10G1200 µs
2x1G12000 µs
NT40A11 4x10G600 µs
4x1G6000 µs
NT40E3-4-PTP 4x10G175 µs
4x1G1750 µs
NT20E3-2-PTP 2x10G350 µs
2x1G3500 µs
NT40A01 4x1G 1750 µs

100% load in the above table is taken as the line rate multiplied by the number of ports. The parameter is inversely proportional to the load percentage, and directly proportional to the average frame size. Thus, if a DeduplicationWindow parameter is desired for 50% load, the above parameters should be multiplied by 2, and so forth.


Example 1: Deduplication

The following NTPL will set up deduplication based on the Layer 3 payload of an IP packet:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Protocol == IP

Example 2: Local retransmission

The following NTPL will cause non-duplicate frames received on port 0 to be retransmitted on port 1. In other words the duplicate frames are dropped:

DeduplicationConfig[retransmit=nonduplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[Streamid=0; DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Example 3: Reusing group IDs

In some cases, it can be beneficial to reuse a group ID for more than one filter with deduplication. As an example of this, one could have a setup where the same ethernet frames are received both in raw form and tunneled. In this case, it could be desirable to calculate a correlation key on the full frame content and the inner content, respectively. This deduplication setup can be created with the following NTPL, where the group ID 0 is reused for both filter expressions:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Example 4: Multiple group IDs

If a scenario requires individual deduplication on multiple ports, i.e. having the ability to receive otherwise identical packets on several ports while having a separate context for each port, multiple group IDs can be used.

The following NTPL configuration can be used to achieve separate deduplication on the L3 payload on ports 1 and 2:

DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1