DeduplicationConfig Command

Reference Documentation

Intel® PAC
Napatech SmartNIC
Reference Information
Napatech Software Suite: DeduplicationConfig Command
DeduplicationConfig Command

The deduplication feature enables an adapter to selectively perform deduplication on traffic based on filter setup.

Internally, the feature uses Correlation Key Recipe to determine whether an incoming packet is a duplicate of a previously received one. Consequently, all protocol offset and masking settings available here can be used to set up the scope of deduplication.

The syntax for the DeduplicationConfig command is as follows:

<DeduplicationConfigAction>     ::= 'DeduplicationConfig' { '[' <DedulicationConfigOptionList> ']' } '=' <DeduplicationGroupTest>
<DeduplicationConfigOptionList> ::= <DeduplicationConfigOption> { ';' <DeduplicationConfigOption> }
<DeduplicationConfigOption>     ::= 'Drop'       '=' ('Duplicate' | 'Nonduplicate') |
                                    'Retransmit' '=' ('Duplicate' | 'Nonduplicate') |
                                    'Colorbit'   '=' ('0' | '1' | ... | '31')
<DeduplicationGroupTest>        ::= 'GroupID' '==' <DeduplicationGroupNumber>

'Drop' will determine which packets are dropped, and can be set to either Duplicate or Nonduplicate
'Retransmit' will determine which packets are retransmitted, and can be set to either Duplicate or Nonduplicate
'Colorbit' will set the provided colorbit on packets considered duplicates.

The 'GroupID' parameter constitutes a second criteria for packets to be considered a match. In other words, both the calulated CorrelationKey, and the GroupID has to match an earlier received packet. Thus, it is up to the specific application whether different or identical GroupIDs are to be used for different filters. A relevant case where identical Group IDs could be employed would be simultaneous matching on tunnelled and untunnelled traffic.

In addition to the settings configured by the DeduplicationConfig command, there is an ini-setting named 'DeduplicationWindow', which determines the expiration period for deduplication. Thus, two packets with matching CorrelationKey and GroupID will have to arrive within this time window in order for the second packet to be considered a duplicate.

Notes on DeduplicationWindow parameter

Owing to the internal architecture of the deduplication feature, the DeduplicationWindow parameter has certain optimal values, depending on the product, the average incoming frame size, and the traffic load. Based on these parameters, the values seen in the table below have been simulated for reference with an average frame size of 300 bytes.

Adapter Line rate Image Window size at 100% load
Intel PAC 4x10G 7000 100 µs
Intel PAC 1x40G 7001 100 µs
NT40A01 4x1G 9500 1600 µs
NT20E3 2x10G 9501 800 µs
NT80E3 2x40G 9503 90 µs
NT200A01 2x40G 9512 90 µs
NT80E3 8x10G 9519 90 µs
NT200A02 2x100G 9521 100 µs
NT200A01 8x10G 9522 90 µs
NT200A02 2x40G 9526 200 µs
NT200A02 8x10G 9533 200 µs
NT40E3 4x10G 9537 100 µs

100% load in the above table is taken as the line rate multiplied by the number of ports. The parameter is inversely proportional to the load percentage, and directly proportional to the average frame size. Thus, if a DeduplicationWindow parameter is desired for 50% load, the above parameters should be multiplied by 2, and so forth.


Example 1: Deduplication

The following NTPL will set up deduplication based on the Layer 3 payload of an IP packet:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Protocol == IP

Example 2: Local retransmission

The following NTPL will cause duplicate frames received on port 0 to be retransmitted on port 1:

DeduplicationConfig[retransmit=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Example 3: Reusing group IDs

In some cases, it can be beneficial to reuse a group ID for more than one filter with deduplication. As an example of this, one could have a setup where the same ethernet frames are received both in raw form and tunneled. In this case, it could be desirable to calculate a correlation key on the full frame content and the inner content, respectively. This deduplication setup can be created with the following NTPL, where the group ID 0 is reused for both filter expressions:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Example 4: Multiple group IDs

If a scenario requires individual deduplication on multiple ports, i.e. having the ability to receive otherwise identical packets on several ports while having a separate context for each port, multiple group IDs can be used.

The following NTPL configuration can be used to achieve separate deduplication on the L3 payload on ports 1 and 2:

DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1