The deduplication feature enables an adapter to selectively perform deduplication on traffic based on filter setup.

Internally, the feature uses Correlation Key Recipe to determine whether an incoming packet is a duplicate of a previously received one. Consequently, all protocol offset and masking settings available here can be used to set up the scope of deduplication.

The syntax for the DeduplicationConfig command is as follows:

<DeduplicationConfigAction>     ::= 'DeduplicationConfig' { '[' <DedulicationConfigOptionList> ']' } '=' <DeduplicationGroupTest>
<DeduplicationConfigOptionList> ::= <DeduplicationConfigOption> { ';' <DeduplicationConfigOption> }
<DeduplicationConfigOption>     ::= 'Drop'       '=' ('Duplicate' | 'Nonduplicate') |
                                    'Retransmit' '=' ('Duplicate' | 'Nonduplicate') |
                                    'Colorbit'   '=' ('0' | '1' | ... | '31')
<DeduplicationGroupTest>        ::= 'GroupID' '==' <DeduplicationGroupNumber>

'Drop' will determine which packets are dropped, and can be set to either Duplicate or Nonduplicate
'Retransmit' will determine which packets are retransmitted, and can be set to either Duplicate or Nonduplicate
'Colorbit' will set the provided colorbit on packets considered duplicates.

The 'GroupID' parameter constitutes a second criteria for packets to be considered a match. In other words, both the calulated CorrelationKey, and the GroupID has to match an earlier received packet. Thus, it is up to the specific application whether different or identical GroupIDs are to be used for different filters. A relevant case where identical Group IDs could be employed would be simultaneous matching on tunnelled and untunnelled traffic.

In addition to the settings configured by the DeduplicationConfig command, there is an ini-setting named 'DeduplicationWindow', which determines the expiration period for deduplication. Thus, two packets with matching CorrelationKey and GroupID will have to arrive within this time window in order for the second packet to be considered a duplicate.

Notes on DeduplicationWindow parameter

Owing to the internal architecture of the deduplication feature, the DeduplicationWindow parameter has certain optimal values, depending on the product, the average incoming frame size, and the traffic load. Based on these parameters, the values seen in the table below have been simulated for reference with an average frame size of 300 bytes.

100% load in the above table is taken as the line rate multiplied by the number of ports. The parameter is inversely proportional to the load percentage, and directly proportional to the average frame size. Thus, if a DeduplicationWindow parameter is desired for 50% load, the above parameters should be multiplied by 2, and so forth.

Examples

Example 1: Deduplication

The following NTPL will set up deduplication based on the Layer 3 payload of an IP packet:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[Streamid=0; CorrelationKey=ckRecipe] = Layer3Protocol == IP

Example 2: Local retransmission

The following NTPL will cause non-duplicate frames received on port 0 to be retransmitted on port 1. In other words the duplicate frames are dropped:

DeduplicationConfig[retransmit=nonduplicate] = GroupID == 0
Define ckRecipe = CorrelationKey(Begin=StartOfFrame[0], End=EndOfFrame[0], DeduplicationGroupID=0)
Assign[Streamid=0; DestinationPort=1; CorrelationKey=ckRecipe] = Port == 0

Example 3: Reusing group IDs

In some cases, it can be beneficial to reuse a group ID for more than one filter with deduplication. As an example of this, one could have a setup where the same ethernet frames are received both in raw form and tunneled. In this case, it could be desirable to calculate a correlation key on the full frame content and the inner content, respectively. This deduplication setup can be created with the following NTPL, where the group ID 0 is reused for both filter expressions:

DeduplicationConfig[drop=duplicate] = GroupID == 0
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=InnerLayer3Header[0], End=InnerLayer3PayloadEnd[0], DeduplicationGroupID=0)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = All
Assign[StreamID=0; CorrelationKey=ckRecipe2] = TunnelType == GREv0

Example 4: Multiple group IDs

If a scenario requires individual deduplication on multiple ports, i.e. having the ability to receive otherwise identical packets on several ports while having a separate context for each port, multiple group IDs can be used.

The following NTPL configuration can be used to achieve separate deduplication on the L3 payload on ports 1 and 2:

DeduplicationConfig[drop=duplicate] = GroupID == 0
DeduplicationConfig[drop=duplicate] = GroupID == 1
Define ckRecipe1 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=0)
Define ckRecipe2 = CorrelationKey(Begin=Layer3Header[0], End=Layer3PayloadEnd[0], DeduplicationGroupID=1)
Assign[StreamID=0; CorrelationKey=ckRecipe1] = Port == 0
Assign[StreamID=1; CorrelationKey=ckRecipe2] = Port == 1