ipfdemo_example.c

Reference Documentation

product_line_custom
Napatech SmartNIC
category
Reference Information

Go to the documentation of this file.

1/* 2 * %NT_SOFTWARE_LICENSE% 3 */ 4 5/** 6 * @example net/ipfdemo/ipfdemo_example.c 7 * @section ipfdemo_example_description Description 8 * 9 * This source file is an example of how to write an IP fragment re-assembling code, 10 * using the Napatech FPGA IPFMode feature. It accelerates IP re-assembling by 11 * making it possible to load balance using a 5 tuple hash algoritm instead of 12 * a 2 tuple hash algorithm. This is done for fragments by learning from the first 13 * fragment and use this knowledge to calculate the 5 tuple for belonging fragments. 14 * The complexing case is the situation where the FPGA was not able to learn 15 * and identify a fragment and therefore it is sent to one of the un-matching streams. 16 * In this situation the code needs to exchange the un-matched packet and deliver 17 * it to the right re-assembling thread. 18 * 19 * 20 * The following NTAPI functions are used: 21 * - @ref NT_Init() 22 * - @ref NT_NetRxOpen() 23 * - @ref NT_NTPL() 24 * - @ref NT_NetRxGet() 25 * - @ref NT_NET_GET_PKT_L2_FRAME_TYPE() 26 * - @ref NT_NET_GET_PKT_L3_FRAME_TYPE() 27 * - @ref NT_NET_GET_PKT_L3_FRAGMENTED() 28 * - @ref NT_NET_GET_PKT_DESCRIPTOR_FORMAT() 29 * - @ref NT_NET_GET_PKT_L3_FIRST_FRAG() 30 * - @ref NT_NET_GET_PKT_L2_PTR() 31 * - @ref NT_NET_GET_PKT_L3_OFFSET() 32 * - @ref NT_NET_GET_SEGMENT_TIMESTAMP() 33 * - @ref NT_NET_GET_PKT_IPF_LAST_FRAGMENT() 34 * - @ref NT_NET_GET_PKT_IPF_UNMATCHED_STREAMID() 35 * - @ref NT_NetRxRelease() 36 * - @ref NT_NetRxClose() 37 * - @ref NT_ExplainError() 38 * 39 * @section ipfdemo_example_prerequisites Prerequisites 40 * A Napatech capture accelerator is needed to run this example with IPFMode support. 41 * The ntservice.ini file must have enough HostBuffersRx defined to honor 42 * the requested streams needed when running this demo application. 43 * Below is an example of a minimum ini-file. It will create 20 64MB RX 44 * hostbuffer from NUMA node 0. 45 * 46 * Note that the PacketDescriptor is set to Ext9, This is needed to run 47 * IPFMode at all. If anything less is specified, then the IPFMode will 48 * not be used and it will revert into a Hash2Tuple mode. 49 * 50 * Note2 that the TimeSyncReferencePriority is set to OsTime. This is critical, when 51 * using a fragtimeout value (default), because the timestamp of each 52 * packet is used to calculate its timeout. 53 * @code 54 * [System] 55 * TimestampFormat = NATIVE_UNIX 56 * 57 * [Adapter0] 58 * AdapterType = NT20E2 59 * BusId = 00:02:00:00 60 * PacketDescriptor = Ext9 61 * HostBuffersRx = [20,64,0] 62 * TimeSyncReferencePriority = OsTime 63 * @endcode 64 * 65 * 66 * ************************* Overview of the algorithm used ****************************** 67 * 68 * The FPGA enables you to match IP fragments and distribute them using a multi CPU 69 * buffer splitting scheme based on a 5-tuple hash algorithm when using the IPFMode feature. 70 * If a multi CPU splitting scheme based on a 2-tuple hash algorithm gives good enough 71 * distribution, then the IPFMode feature in the FPGA should not be used. All belonging 72 * fragments will end up in same stream using a 2-tuple hash algorithm. 73 * IPFMode: how it works: 74 * - Learns from first fragment 75 * - First fragment contains streamID of unmatched fragments 76 * - Timeout setup for releasing learned entry 77 * 78 * @code 79 * Graphical overview: 80 * 81 * +--------------+ 82 * | DOI hash | 83 * +------>| |------+ 84 * | | tables | | 85 * | +--------------+ | 86 * | | 87 * | | 88 * +----------------+ | | +----------------+ 89 * | |------+ +----->| | 90 * | Reassembling | +--------------+ | Unmatched | 91 * | | | Return | | | 92 * | threads |------------->| msg box |------------>| threads | 93 * | | | FIFOs | | | 94 * | |<-----+ +--------------+ +------| | 95 * +----------------+ | | +----------------+ 96 * ^ ^ ^ | | ^ ^ 97 * | | | | | | | 98 * | | | | | | | 99 * | | | | +--------------+ | | | 100 * | | | | | Msg box | | | | 101 * | | | +-------| |<-----+ | | 102 * | | | | FIFOs | | | 103 * | | | +--------------+ | | 104 * | | | | | 105 * 5 tuple hash splitting 2 tuple hash splitting 106 * | | | | | 107 * | | | | | 108 * +-------------------------------------------------------------------------------+ 109 * | | 110 * | NTAPI/FPGA | 111 * | | 112 * +-------------------------------------------------------------------------------+ 113 * 114 * @endcode 115 * 116 * Description of algorithm: 117 * 118 * 119 * Threads: 120 * -------- 121 * 122 * Unmatched threads – variable number (N) – number of unmatched streams to use 123 * (2-tuple hash cpu-splitting) 124 * 125 * Each unmatched thread receives the unmatched fragments. When a fragment is received, 126 * it is checked in the DOI tables to find a datagram ID match. If a match is found, the 127 * NetBuf containing the fragment is send to that reassembling Msg box FIFO, of which 128 * the DOI table belongs to. Otherwise, if no match found (yet), the fragment is 129 * temporarily put into a wait list. The wait list is periodically checked against the 130 * DOI tables. 131 * 132 * Reassembling threads – variable number (M) – number of streams to use 133 * (5-tuple hash cpu-splitting) 134 * 135 * When each reassembling thread receives IP fragments, it collects them into complete 136 * datagrams. If all fragments are received in order, or at least the first fragment is 137 * received first, then the need for unmatched fragment collection is not needed and all 138 * datagrams are reassembled in the reassembling threads. 139 * When the reassembling thread receives a first fragment, it then stores that fragment 140 * in its local collection hash table (tbl) and makes that datagram ID available for the 141 * specific unmatched thread to inform about its interest in fragments belonging to this 142 * datagram, using the specific DOI table connecting the reassembling thread with the 143 * unmatched thread. 144 * When all fragments are received for a datagram, they are all released and the DOI 145 * table entry is cleared. 146 * Each reassembling thread first reads from the Msg box for a NetBuf, and if none found, 147 * then it reads the NTAPI stream for a NetBuf. If any fragments are received from here, 148 * they are put into the collection table (tbl). 149 * 150 * 151 * Inter-thread communication messages: 152 * ------------------------------------ 153 * 154 * Msg boxes (FIFOs): 155 * 156 * Used by Unmatched threads to send unmatched fragments to the reassembling threads. 157 * [[[unm1 Msgbox],[ unm2 Msgbox],…[ unmN Msgbox]], [[unm1 Msgbox],[ unm2 Msgbox],…[ unmN Msgbox]]…M times] 158 * reasm1 reasm2 …reasmM 159 * 160 * The Msg boxes are built as FIFOs and are used by the unmatched threads to send the 161 * NetBuf of a received datagram fragment to the belonging reassembling thread. 162 163 * Return Msg boxes (FIFOs): 164 * 165 * Used by Reassmbling threads to send unmatched-fragment NetBufs back to the unmatched 166 * threads originally received them. 167 * [[[reasm1 Msgbox],[ reasm2 Msgbox],…[ reasmM Msgbox]], [[reasm1 Msgbox],[ reasm2 Msgbox],…[ reasmN Msgbox]]…N times] 168 * unm1 unm2 …unmN 169 * 170 * This FIFO is only needed to make the complete algorithm lockless. The NTAPI cannot 171 * handle multi-threaded access to the packet interface without serialization. 172 173 * DOI tables (Datagram of interest): 174 * 175 * The DOI tables are a set of hash tables containing a list of datagram IDs that the 176 * reassembling threads are interested in. 177 * 178 * The FPGA specifies the unmatched stream ID together with the reception of the first 179 * fragment. This information is used by the reassembling threads to inform the specific 180 * unmatched thread about where to send unmatched fragments matching the first fragments 181 * datagram ID. 182 * 183 * Each reassembling thread has one DOI table allocated for each unmatched thread, thus 184 * M*N DOI hash tables are used. This way each reassembling thread has exclusive write 185 * access to one table for each unmatched tread. 186 * [[[reasm1 tbl],[reasm2 tbl],…[reasmM tbl]], [[reasm1 tbl],[reasm2 tbl],…[reasmM tbl]]…N times] 187 * Unmatched1 Unmatched2 …UnmatchedN 188 * 189 * Each unmatched thread only reads information from the DOI tables. When an unmatched 190 * fragment matches an entry in the DOI table, then this fragment is send to the 191 * reassembling thread using the dedicated Msg box FIFO. 192 * 193 * Datagram ID calculation: 194 * 195 * The fragments belonging to a datagram are identified by the calculated datagram ID: 196 * It consists of information from 4 fields in the IP header (source IP, destination IP, 197 * Identification and protocol). 198 199 *<hr> 200 * @section ipfdemo_example_code Code 201 * @} 202 */ 203 204#if defined(__linux__) || defined(__FreeBSD__) 205 #include <unistd.h> 206 #include <signal.h> 207 #include <assert.h> 208 #include <pthread.h> 209 #include <arpa/inet.h> 210 #include <sys/time.h> 211#elif defined(WIN32) || defined (WIN64) 212 #include <winsock2.h> // ntohs() 213 #include <time.h> 214 #include <sys/timeb.h> 215 #include <process.h> // threading 216#endif 217 218#include <argparse.h> 219#include <nt.h> 220 221 222#if defined(WIN32) || defined (WIN64) 223 224 //#define snprintf _snprintf 225 #define snprintf(a, b, c, d) _snprintf_s((a), _countof(a), (b), (c), (d)) 226 #define strcpy(s, a) strcpy_s((s), _countof(s), (a)) 227 228 // Sleep 229 void sleep(int time) 230 { 231 Sleep(time * 1000); /* From seconds to milliseconds */ 232 } 233 234 void usleep(unsigned long usec) 235 { 236 Sleep(usec / 1000); /* From useconds to milliseconds */ 237 } 238 239 // Time 240 struct timezone { 241 int tz_minuteswest; /* minutes west of Greenwich */ 242 int tz_dsttime; /* type of DST correction */ 243 }; 244 245 int gettimeofday(struct timeval *tv, struct timezone *tz) 246 { 247 struct __timeb64 timebuffer; 248 _ftime64_s(&timebuffer); 249 tv->tv_sec = (long)timebuffer.time; 250 tv->tv_usec = timebuffer.millitm * 1000; 251 return 0; 252 } 253 254 // Threading 255 typedef HANDLE pthread_t; 256 typedef unsigned (__stdcall *start_address_t)(void *parameter); 257 258 int pthread_create(HANDLE *thread, DWORD *attr, start_address_t start_routine, void *parameter) 259 { 260 HANDLE handle = (HANDLE)_beginthreadex( NULL, 0, start_routine, parameter, 0, NULL); 261 if (handle == 0) 262 { 263 int status = GetLastError(); 264 fprintf(stderr, "pthread_create() fail with error: 0x%x\n", status); 265 *thread = NULL; 266 return status | NT_SYSTEM_ERRORS; 267 } 268 269 *thread = handle; 270 return NT_SUCCESS; 271 } 272 273 int pthread_join(HANDLE thread, void **value_ptr) 274 { 275 WaitForSingleObject(thread, INFINITE); 276 CloseHandle(thread); 277 return NT_SUCCESS; 278 } 279 280#endif 281 282#pragma pack(push, 1) 283/* IPv4 header structure */ 284typedef struct iphdr_s { 285 uint8_t ipHlen:4; // little endian layout 286 uint8_t ipVer:4; // little endian layout 287 uint8_t tos; 288 uint16_t ipTotlen; 289 uint16_t Id; 290#define FRAG_OFFS_MASK 0x1FFF 291#define LAST_FRAG_BITS 0x00E0 292 uint16_t fragOffs; 293 uint8_t ttl; 294 uint8_t prot; 295 uint16_t csum; 296 uint32_t srcAddr; 297 uint32_t dstAddr; 298} iphdr_t; 299#pragma pack(pop) 300 301#define L3_ADDR(_NetBuf_) ((uint8_t *)NT_NET_GET_PKT_L2_PTR(_NetBuf_)+NT_NET_GET_PKT_L3_OFFSET(_NetBuf_)) 302#define IPVERSION(_NetBuf_) (((iphdr_t *)L3_ADDR(_NetBuf_))->ipVer) 303 304/* macros for IPv4 packets */ 305#define IPV4_HDR_LENGTH(_NetBuf_) ((uint8_t)(((iphdr_t *)L3_ADDR(_NetBuf_))->ipHlen)<<2) 306#define IPV4_TOT_LEN(_NetBuf_) (ntohs(((iphdr_t *)L3_ADDR(_NetBuf_))->ipTotlen)) 307#define IPV4_FRAGMENT_ID(_NetBuf_) (ntohs(((iphdr_t *)L3_ADDR(_NetBuf_))->Id)) 308#define IPV4_FRAGMENT_OFFSET(_NetBuf_) ((ntohs(((iphdr_t *)L3_ADDR(_NetBuf_))->fragOffs)&FRAG_OFFS_MASK)<<3) 309#define IPV4_LAST_FRAG(_NetBuf_) (((((iphdr_t *)L3_ADDR(_NetBuf_))->fragOffs)&LAST_FRAG_BITS)==0) 310#define IPV4_SRC_ADDR(_NetBuf_) (((iphdr_t *)L3_ADDR(_NetBuf_))->srcAddr) 311#define IPV4_DST_ADDR(_NetBuf_) (((iphdr_t *)L3_ADDR(_NetBuf_))->dstAddr) 312#define IPV4_PROTOCOL(_NetBuf_) (((iphdr_t *)L3_ADDR(_NetBuf_))->prot) 313 314#define IPV4_GET_DGRAM_ID(_NetBuf_, _src_, _dst_, _id_, _prot_) \ 315 do {iphdr_t *_ip_=(iphdr_t *)L3_ADDR(_NetBuf_);_src_=_ip_->srcAddr;_dst_=_ip_->dstAddr;\ 316 _id_=_ip_->Id;_prot_=_ip_->prot;}while(0) 317 318#define IPV4_DATA_LEN(_NetBuf_) (IPV4_TOT_LEN(_NetBuf_)-IPV4_HDR_LENGTH(_NetBuf_)) 319 320 321/* Help structure to concatenate ID source fields */ 322typedef union _DOI_dgramId_u { 323 volatile uint64_t src_id; 324 struct { 325 volatile uint32_t src; 326 volatile uint32_t id; 327 }; 328} DOI_dgramId_u; 329 330#define MAX_SRC_ID 8 331typedef struct _DOI_dgramTbl_t { 332 volatile uint64_t aSrc_id[MAX_SRC_ID]; // Source and frag ID concatenated into ID1 333 volatile uint64_t aDst_pr[MAX_SRC_ID]; // Destination and protocol concatenated into ID2 334} DOI_dgramTbl_t; 335 336#define DOI_FRAG_TBL_SIZE 8081 /* Prime number */ 337#define DOI_HASH_GET_KEY(_entry_, _src_, _id_) (_entry_ = (_src_+_id_)%DOI_FRAG_TBL_SIZE) 338 339 340/* Re-assembler hash table types and definitions */ 341struct frag_s { 342 NtNetBuf_t hNetBuf; // hNetBuf containing fragment buffer 343 uint16_t offset; // Fragment offset 344 uint16_t size; // Size of this fragment 345 uint8_t firstFrag; // If marked as first fragment 346 uint8_t lastFrag; // If marked as last fragment 347 uint8_t fromUnm; // Is this packet received from an un-matched thread (needed to lock on release) 348 uint8_t unmIndex; // if from unmatched stream, then this tells which unmatched thread index 349}; 350 351#define MAX_FRAG_CNT 18 352#define REASSEMBLY_HASH_TBL_SIZE 1021 /* Prime number */ 353#define INITIAL_HASH_TBL_ENTRY_CNT 1024 354typedef struct _hashTbl_entry_s { 355 uint64_t id1; // Src and prot id for ID1 356 uint64_t id2; // Dest and prot Id for ID2 357 volatile uint64_t *pDOI_Src_id; // Source and frag ID field 358 volatile uint64_t *pDOI_Dst_pr; // Dest and protocol ID field 359 uint16_t fragCnt; // Number of fragments in aFrag 360 struct frag_s aFrag[MAX_FRAG_CNT]; // Array of all fragments belonging to the same datagram 361 struct _hashTbl_entry_s *pNext; // Pointer to next element in list 362} tbl_entry_t; 363 364/* Hash table macros */ 365#define LOOKUP_ENTRY(_tbl_base_, _tbl_entry_type_, _tbl_size_, _id1_, _id2_, _tbl_entry_) \ 366 {uint32_t _key_=(uint32_t)((_id1_^_id2_)%_tbl_size_);_tbl_entry_type_ *_tbl_; \ 367 _tbl_ = _tbl_base_[_key_]; while(_tbl_ && (_tbl_->id1 != _id1_ || _tbl_->id2 != _id2_)) _tbl_ = _tbl_->pNext; \ 368 _tbl_entry_=_tbl_;} 369 370#define GET_NEW_TBL_ENTRY(_tbl_free_, _tbl_entry_type_, _tbl_entry_) \ 371{if (!_tbl_free_){int __o__;_tbl_entry_type_ *__plm__; \ 372 for(__o__=0;__o__<64;__o__++){__plm__=calloc(1,sizeof(_tbl_entry_type_)); \ 373 if (!__plm__) assert(0); __plm__->pNext=_tbl_free_;_tbl_free_=__plm__;}} \ 374 _tbl_entry_ = _tbl_free_;_tbl_free_ = _tbl_free_->pNext;tbl_entry->pNext = NULL;} 375 376#define RELEASE_TBL_ENTRY(_tbl_free_, _tbl_entry_) \ 377 {_tbl_entry_->pNext = _tbl_free_; _tbl_free_ = _tbl_entry_;} 378 379#define ADD_ENTRY_TO_TBL(_tbl_base_, _tbl_entry_type_, _tbl_size_, _tbl_entry_) \ 380 {uint32_t _key_=(uint32_t)((_tbl_entry_->id1^_tbl_entry_->id2)%_tbl_size_);_tbl_entry_type_ *_tbl_; \ 381 _tbl_ = _tbl_base_[_key_]; while (_tbl_ && (_tbl_->id1 != _tbl_entry_->id1 || _tbl_->id2 != _tbl_entry_->id2)) _tbl_ = _tbl_->pNext; \ 382 if (_tbl_ == NULL) {_tbl_entry_->pNext = _tbl_base_[_key_]; _tbl_base_[_key_]=_tbl_entry_;} else assert(_tbl_==_tbl_entry_);} 383 384#define DEL_ENTRY_FROM_TBL(_tbl_base_, _tbl_entry_type_, _tbl_size_, _tbl_entry_) \ 385 {uint32_t _key_=(uint32_t)((_tbl_entry_->id1^_tbl_entry_->id2)%_tbl_size_);_tbl_entry_type_ *_tbl_, *_prev_ = NULL; \ 386 _tbl_ = _tbl_base_[_key_]; \ 387 while (_tbl_ && (_tbl_->id1 != _tbl_entry_->id1 || _tbl_->id2 != _tbl_entry_->id2)) { \ 388 _prev_ = _tbl_;_tbl_ = _tbl_->pNext;} \ 389 if (_tbl_) {if (_prev_) _prev_->pNext = _tbl_->pNext;else _tbl_base_[_key_] = _tbl_->pNext;}else assert(0);} 390 391 392typedef struct _ipDefrag ipDefrag_t; 393 394 395#if defined(__linux__) || defined(__FreeBSD__) 396 #define compiler_barrier() __asm__ __volatile__("":::"memory") 397#elif defined(WIN32) || defined (WIN64) 398 #define compiler_barrier() 399#endif 400 401 402/* Msg box definition and control type and macros */ 403#define MSG_BOX_DEPTH 0x2000 404typedef struct _msg_box_s { 405 volatile uint32_t rdIdx; // Read index of msg box 406 volatile uint32_t wrIdx; // Write index of msg box 407 NtNetBuf_t data[MSG_BOX_DEPTH]; // Msg box elements (pointers to hNetBufs) 408} msg_box_t; 409 410/* Macros to use the msg box */ 411#define MSG_BOX_EMPTY(_fifo_) ((_fifo_)->wrIdx==(_fifo_)->rdIdx) 412#define MSG_BOX_FULL(_fifo_) (((_fifo_)->wrIdx-(_fifo_)->rdIdx) >= MSG_BOX_DEPTH) 413#define MSG_BOX_GET(_fifo_) (_fifo_)->data[(_fifo_)->rdIdx&(MSG_BOX_DEPTH-1)]; compiler_barrier(); (_fifo_)->rdIdx++ 414#define MSG_BOX_PUT(_fifo_, _elem_) \ 415 do { \ 416 (_fifo_)->data[(_fifo_)->wrIdx&(MSG_BOX_DEPTH-1)]=_elem_ ; \ 417 compiler_barrier(); \ 418 (_fifo_)->wrIdx++; \ 419 } while (0) 420 421 422 423/* Wait-list type definition */ 424#define MAX_WAIT_ELEM 1024 425typedef struct wait_list_s { 426 NtNetBuf_t hNetBuf; 427 struct wait_list_s *pNext; 428} wait_list_t; 429 430/* Un-matched thread variable definition */ 431typedef struct _unmThread_s { 432 pthread_t thread; // This thread 433 int streamId; // NT Stream Id to service by this thread 434 DOI_dgramTbl_t *pDOI; // Datagram Of Interest tables. For this Unmatched Thread only, indexed for each Reassembly Thread 435 msg_box_t *pMsgboxReturn; // All msg boxes used by the current unmatched thread indexed [reasmCnt] 436 wait_list_t WaitListElem[MAX_WAIT_ELEM]; // All awailable wait-list elements 437 wait_list_t *pFree; // Linked list of free wait-list elements 438 wait_list_t *pWait; // Linked list of waiting wait-list elements 439 ipDefrag_t *pIpDefrag; // Pointer to main instance variable 440 /* Statistics */ 441 int fragRecv; // Counter for all fragments received by this un-matched thread 442 int fragsDeleted; // Counter for all timed out and deleted fragments in wait-list 443} unmThread_t; 444 445 446/* Re-assembling thread variable definition */ 447typedef struct _reasmThread_s { 448 pthread_t thread; // This thread 449 int idx; // Index number of this re-assembling thread (first re-assembling thread starts with index 0) 450 int streamId; // NT Stream Id to service by this thread 451 tbl_entry_t *tbl[REASSEMBLY_HASH_TBL_SIZE]; // Hash table for the current re-assembling thread 452 tbl_entry_t *tblFree; // Free elements for the hash table 453 msg_box_t *pMsgbox; // All msg boxes used by the current re-assembling thread indexed [unmCnt] 454 ipDefrag_t *pIpDefrag; // Pointer to main instance variable 455 /* statistics */ 456 int datagramCompleted; // Counter for datagrams sent in fragments and re-assembled sucessfully 457 int fragmentsTimedout; // Counter for all timed out and deleted fragments in hash table 458 int firstFragRcv; // Counter for all first-fragments received by this re-assembling thread 459 int nonFragments; // Counter for all non-fragmented packets received by this re-assembling stream 460 int Src_IdClash; // Counter for all Id clashes encountered (frags with src,dst,prot,fragId identical) 461 int msgboxPackets; // Counter for remaining msg box entries on exit 462} reasmThread_t; 463 464struct _ipDefrag { 465 int Running; // Application running control variable 466 int adapterNo; // Adapter number to run on 467 int unmCnt; // number of un-matched streams 468 int unmStart; // Value of first un-matched streamid 469 int reasmCnt; // Number of re-assembling streams 470 int reasmStart; // Value of first re-assembling streamId 471 uint64_t fragTimeout; // Fragment timeout setting in ms 472 unmThread_t *pUnmThreads; // All un-matched thread instance pointers 473 DOI_dgramTbl_t *pDOI_Tbls; // Hash tables to notify interest of fragments to the Unmatched Threads - indexed [unmCnt][reasmCnt] 474 reasmThread_t *pReasmThreads; // All re-assembling thread instance pointers 475 msg_box_t *pReasmMsgboxes; // Pointer to all re-assembling threads message boxes 476 msg_box_t *pReasmReturnMsgboxes; // Pointer to all NetBuf return message boxes 477 int ExtDescrType; // The NT extended descriptor type used by FPGA to access and use the correct macros 478 int allReasmClosed; // To let all Unm-threads/streams close after re-assembling threads are closed 479}; 480 481 482#define NO_OPTIONS 0 483#define OPTION_HELP (1<<1) 484#define OPTION_ADAPTER (1<<2) 485#define OPTION_NUM_REASM (1<<3) 486#define OPTION_NUM_UNM (1<<4) 487#define OPTION_TIMEOUT (1<<5) 488#define OPTION_TABLE_PERSIST (1<<6) 489#define OPTION_TABLE_TIMEOUT (1<<7) 490 491static int opt_adapter = -1; 492static int opt_reasm = -1; 493static int opt_unm = -1; 494static int opt_frag = -1; 495static char *opt_persist = NULL; 496static int opt_timeout = -1; 497 498/** 499 * Table of valid options. 500 */ 501struct argparse_option arg_options[] = { 502 OPT_HELP(), 503 OPT_INTEGER('a', "adapter", &opt_adapter, "The adapter to run tests on", NULL, 0, 0, "adapter number"), 504 OPT_INTEGER('r', "reasm", &opt_reasm, "Number of concurrent IP fragments re-assembling streams/threads", NULL, 0, 0, "number"), 505 OPT_INTEGER('u', "unm", &opt_unm, "Number of concurrent un-matched streams/threads", NULL, 0, 0, "number"), 506 OPT_INTEGER('f', "fragtimeout", &opt_frag, "How old fragments may get before they are deleted from tables (ms)", NULL, 0, 0, "ms"), 507 OPT_STRING( 'p', "tablepersist", &opt_persist, "FPGA TablePersist", NULL, 0, 0, "timeout|lastfragment"), 508 OPT_INTEGER('t', "tabletimeout", &opt_timeout, "FPGA table timeout", NULL, 0, 0, "value"), 509 OPT_END(), 510}; 511 512static struct _ipDefrag IpDefrag; // The main instance handle 513 514static const char *usageText[] = { 515 "Syntax:\n" 516 "\n" 517 "ipfdemo_example [-a <adapter number>] [-r <number>] [-u <number>] [-f <ms>] [-p <timeout|lastfragment>] [-t <value>]\n" 518 "\nCommands:\n", 519 NULL}; 520 521/***************************************************************************** 522 Function to get system time in milliseconds 523******************************************************************************/ 524static uint64_t GetSystemTimeNoArg(void) 525{ 526 struct timeval tv; 527 gettimeofday(&tv, NULL); 528 return tv.tv_sec * 1000000 + tv.tv_usec; 529} 530 531/***************************************************************************** 532 Called from the un-matched threads, to check if any re-assembling threads 533 are waiting for this fragment. It is communicated using the DOI hash table. 534 One table for each re-assembling/un-matched thread combination. This is done 535 to make singular write access to each table (no locks needed). 536******************************************************************************/ 537static int SendToReasm(unmThread_t *pUnm, NtNetBuf_t *phNetBuf) 538{ 539 int i; 540 int32_t entry; 541 uint32_t src, dst; 542 uint8_t prot; 543 uint16_t id; 544 DOI_dgramId_u val, val1; 545 DOI_dgramTbl_t *pTbl; 546 msg_box_t *pMsgbox; 547 int idx; 548 NtNetBuf_t hNetBuf = *phNetBuf; 549 550 IPV4_GET_DGRAM_ID(hNetBuf, src, dst, id, prot); 551 552 val.src = src; 553 val.id = id; 554 val1.src = dst; 555 val1.id = prot; 556 557 for (i = 0; i < pUnm->pIpDefrag->reasmCnt; i++) { 558 pTbl = &pUnm->pDOI[i * DOI_FRAG_TBL_SIZE]; 559 DOI_HASH_GET_KEY(entry, src, id); 560 pMsgbox = &pUnm->pIpDefrag->pReasmMsgboxes[i * pUnm->pIpDefrag->unmCnt + (pUnm->streamId - pUnm->pIpDefrag->unmStart)]; 561 562 idx=0; 563 while (idx < MAX_SRC_ID) { 564 if (pTbl[entry].aSrc_id[idx] == val.src_id && 565 pTbl[entry].aDst_pr[idx] == val1.src_id) { 566 if (!MSG_BOX_FULL(pMsgbox)) { 567 MSG_BOX_PUT(pMsgbox, hNetBuf); 568 *phNetBuf = NULL; 569 return 1; 570 } else { 571 /* Msgbox full, we try again later */ 572 return 0; 573 } 574 } 575 idx++; 576 } 577 } 578 return 0; 579} 580 581/***************************************************************************** 582 This is the main un-matched thread routine. One thread for each un-matched 583 stream is spawned to handle all the un-matched streams configured. 584 Upon receival of an IP fragment from hardware, this routine searches the DOI 585 tables (restricted to the ones associated with this stream ID) for interest 586 in this fragment from any re-assembling threads. If one is waiting for fragments 587 belonging to this particular datagram, it is send to that re-assembling thread 588 using the msg box mechanism. If none interested (yet), the fragment is temporarily 589 stored in a wait-list. Frequently the wait-list is scanned together with the DOI 590 tables to make sure the fragments are sent to the corresponding re-assembling 591 thread when it raises interest herein. 592 If the wait-list is full, no more fragments are read until the last hNetBuf 593 can be delivered or added to wait-list. 594******************************************************************************/ 595#if defined(__linux__) || defined(__FreeBSD__) 596 static void *_UnMatchedThread(void *arg) 597#elif defined(WIN32) || defined (WIN64) 598 static unsigned __stdcall _UnMatchedThread(void *arg) 599#endif 600{ 601 unmThread_t *pUnm = (unmThread_t *)arg; // this pointer for this thread 602 char streamName[20]; // Stream name 603 char errorBuffer[NT_ERRBUF_SIZE]; // Error buffer 604 NtNetStreamRx_t hNetRx; // Handle to the RX stream 605 NtNetBuf_t hNetBuf = NULL; // Net buffer container 606 int i; // loop variable 607 int status; // status from NTAPI calls 608 int invalidPkt; // Invalid packet indicator (we only receive IP fragments with these streams) 609 int checkTimeout = 0; // Indicator for fragment timeout check 610 uint64_t sysTs = 0; // System Timestamp 611 int deleteIt; // If we should delete the entry from the wait-list. Is delivered or timed out 612 wait_list_t *pWait; // Help pointer to cleanup wait list on exit 613 614 sprintf(streamName,"IPFUnm_%i", pUnm->streamId); 615 616 /* Fill free list with elements for the wait-list */ 617 pUnm->WaitListElem[0].pNext = NULL; 618 for (i = 1; i < MAX_WAIT_ELEM; i++) { 619 pUnm->WaitListElem[i].pNext = &pUnm->WaitListElem[i - 1]; 620 } 621 pUnm->pFree = &pUnm->WaitListElem[MAX_WAIT_ELEM - 1]; 622 623 // Get a stream handle 624 if((status = NT_NetRxOpen(&hNetRx, streamName, NT_NET_INTERFACE_PACKET, pUnm->streamId, -1)) != NT_SUCCESS) { 625 // Get the status code as text 626 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 627 fprintf(stderr, "[%i] NT_NetRxOpen() failed: %s\n", pUnm->streamId, errorBuffer); 628#if defined(__linux__) || defined(__FreeBSD__) 629 return (void *)NULL; 630#elif defined(WIN32) || defined (WIN64) 631 { _endthreadex(0); return 0; } 632#endif 633 } 634 635 while(pUnm->pIpDefrag->Running) { 636 637 /* Check all Return Msg boxes for fragment packets to be freed */ 638 for (i = 0; i < pUnm->pIpDefrag->reasmCnt; i++) { 639 while (!MSG_BOX_EMPTY(&pUnm->pMsgboxReturn[i])) { 640 NtNetBuf_t hNetBuf1; 641 hNetBuf1 = MSG_BOX_GET(&pUnm->pMsgboxReturn[i]); 642 // printf("Releasing unmatched fragment %i (%llx) by thread index %i\n", i, hNetBuf1, pUnm->streamId - pUnm->pIpDefrag->unmStart); 643 NT_NetRxRelease(hNetRx, hNetBuf1); 644 } 645 } 646 647 /* Check for entries in the wait-list to be delivered to re-assembling thread who has signaled interest through (DOI hash table) */ 648 if (pUnm->pWait) { 649 wait_list_t *pNxt, *pPrev = NULL, *pWaitFrag = pUnm->pWait; 650 if (checkTimeout) { 651 sysTs = GetSystemTimeNoArg()*100; // make it 10ns units 652 } 653 /* Loop all elements in list */ 654 while (pWaitFrag) { 655 /* Try to send to an interested re-assembly tread */ 656 deleteIt = SendToReasm(pUnm, &pWaitFrag->hNetBuf); 657 if (!deleteIt && checkTimeout) { 658 /* check for timed out condition */ 659 if (sysTs - NT_NET_GET_PKT_TIMESTAMP(pWaitFrag->hNetBuf) > pUnm->pIpDefrag->fragTimeout) { 660 deleteIt=1; 661 } 662 } 663 /* Remove from list if send or timed out */ 664 if (deleteIt) { 665 if (pWaitFrag->hNetBuf) { 666 NT_NetRxRelease(hNetRx, pWaitFrag->hNetBuf); 667 pWaitFrag->hNetBuf = NULL; 668 pUnm->fragsDeleted++; 669 } 670 pNxt = pWaitFrag->pNext; 671 /* Move base if needed */ 672 if (pUnm->pWait == pWaitFrag) { 673 pUnm->pWait = pNxt; 674 } else 675 if (pPrev) { 676 pPrev->pNext = pWaitFrag->pNext; 677 } 678 /* Put onto free list */ 679 pWaitFrag->pNext = pUnm->pFree; 680 pUnm->pFree = pWaitFrag; 681 /* Step to next element */ 682 pWaitFrag = pNxt; 683 } else { 684 /* Jump over element */ 685 pPrev = pWaitFrag; 686 pWaitFrag = pWaitFrag->pNext; 687 } 688 } 689 } 690 691 if (hNetBuf == NULL) { 692 /* Read new fragments from stream */ 693 if((status = NT_NetRxGet(hNetRx, &hNetBuf, 1)) != NT_SUCCESS) { 694 if((status == NT_STATUS_TIMEOUT) || (status == NT_STATUS_TRYAGAIN)) { 695 if (pUnm->pIpDefrag->fragTimeout) checkTimeout = 1; 696 continue; 697 } 698 // Get the status code as text 699 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 700 fprintf(stderr, "[%i] NT_NetRxGet() failed: %s\n", pUnm->streamId, errorBuffer); 701 #if defined(__linux__) || defined(__FreeBSD__) 702 return (void *)NULL; 703 #elif defined(WIN32) || defined (WIN64) 704 { _endthreadex(0); return 0; } 705 #endif 706 } 707 } 708 709 assert(hNetBuf != NULL); 710 711 /* New packet received */ 712 invalidPkt = 1; 713 714 if (NT_NET_GET_PKT_L2_FRAME_TYPE(hNetBuf) == NT_L2_FRAME_TYPE_ETHER_II && 715 NT_NET_GET_PKT_L3_FRAME_TYPE(hNetBuf) == NT_L3_FRAME_TYPE_IPv4 && 716 NT_NET_GET_PKT_L3_FRAGMENTED(hNetBuf)) { 717 718 /* Packet is fragmented and valid for our use */ 719 if (NT_NET_GET_PKT_DESCRIPTOR_FORMAT(hNetBuf) == 9) { 720 if (NT_NET_GET_PKT_L3_FIRST_FRAG(hNetBuf)) { 721 fprintf(stderr, "[%i] ERROR un-matched streams may never receive first fragment\n", pUnm->streamId); 722 exit(0); 723 } 724 725 invalidPkt = 0; 726 if (SendToReasm(pUnm, &hNetBuf) == 0) { 727 /* No match found. Put entry into wait list */ 728 if (pUnm->pFree == NULL) { 729 /* We wait for available slots */ 730 usleep(1); 731 if (pUnm->pIpDefrag->fragTimeout) checkTimeout = 1; 732 } else { 733 wait_list_t *pNxt; 734 pNxt = pUnm->pFree->pNext; 735 pUnm->pFree->pNext = pUnm->pWait; 736 pUnm->pWait = pUnm->pFree; 737 pUnm->pFree = pNxt; 738 pUnm->pWait->hNetBuf = hNetBuf; 739 /* Buffer deliverd to wait-list */ 740 hNetBuf = NULL; 741 } 742 } 743 744 if (hNetBuf == NULL) { 745 pUnm->fragRecv++; 746 } 747 } 748 } 749 750 if (invalidPkt && !NT_NET_GET_PKT_L3_FRAGMENTED(hNetBuf)) { 751 fprintf(stderr, "[%i] ERROR, invalid non-ipv4-fragment received on one un-matched fragment stream\n", pUnm->streamId); 752 exit(0); 753 } 754 } 755 756/* Exiting section */ 757 pWait = pUnm->pWait; 758 while (pWait) { 759 // Release pending fragment packets 760 NT_NetRxRelease(hNetRx, pWait->hNetBuf); 761 pWait = pWait->pNext; 762 } 763 764 /* All re-assembling threads has to close before we may close the un-matched threads. 765 We need these threads to continue running while freeing hNetBuf's sent from the un-matched streams */ 766 while (pUnm->pIpDefrag->allReasmClosed == 0) { 767 usleep(1000); 768 } 769 770 /* Empty all Return Msg Boxes */ 771 for (i = 0; i < pUnm->pIpDefrag->reasmCnt; i++) { 772 while (!MSG_BOX_EMPTY(&pUnm->pMsgboxReturn[i])) { 773 NtNetBuf_t buf = MSG_BOX_GET(&pUnm->pMsgboxReturn[i]); 774 NT_NetRxRelease(hNetRx, buf); 775 } 776 } 777 778 /* Close NT stream */ 779 if ((status = NT_NetRxClose(hNetRx)) != NT_SUCCESS) { 780 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 781 fprintf(stderr, "[%i] Un-matched: NT_NetRxGet() failed: %s\n", pUnm->streamId, errorBuffer); 782 } 783 784#if defined(__linux__) || defined(__FreeBSD__) 785 return (void *)NULL; 786#elif defined(WIN32) || defined (WIN64) 787 { _endthreadex(0); return 0; } 788#endif 789} 790 791 792 793static void _FreeUnmFragment(reasmThread_t *pReasm, int unmIndex, NtNetBuf_t hNetBuf) 794{ 795 msg_box_t *pMsgbox; 796 int waitCnt; 797 pMsgbox = &pReasm->pIpDefrag->pReasmReturnMsgboxes[unmIndex * pReasm->pIpDefrag->reasmCnt + (pReasm->streamId - pReasm->pIpDefrag->reasmStart)]; 798 799 waitCnt=0; 800 while (waitCnt < 100) { 801 if (MSG_BOX_FULL(pMsgbox)) { 802 usleep(1000); 803 waitCnt++; 804 } else { 805 MSG_BOX_PUT(pMsgbox, hNetBuf); 806 break; 807 } 808 }; 809 810 if (waitCnt == 100) { 811 fprintf(stderr, "[%i] Return Msg Box full - failed to put element.\n", pReasm->streamId); 812 } 813} 814 815 816 817/***************************************************************************** 818 Free a complete re-assembled datagram 819******************************************************************************/ 820static void _ReleaseDgramTblEntry(reasmThread_t *pReasm, tbl_entry_t *tbl_entry, NtNetStreamRx_t hNetRx) 821{ 822 int i; 823 DEL_ENTRY_FROM_TBL(pReasm->tbl, tbl_entry_t, REASSEMBLY_HASH_TBL_SIZE, tbl_entry); 824 825 for (i = 0; i < tbl_entry->fragCnt; i++) { 826 if (tbl_entry->aFrag[i].fromUnm) { 827 /* Find Return Msg box to return NetBuf to */ 828 _FreeUnmFragment(pReasm, tbl_entry->aFrag[i].unmIndex, tbl_entry->aFrag[i].hNetBuf); 829 } else { 830 NT_NetRxRelease(hNetRx, tbl_entry->aFrag[i].hNetBuf); 831 } 832 } 833 834 /* zero out interest in DOI tbl */ 835 if (tbl_entry->pDOI_Src_id) { 836 *tbl_entry->pDOI_Src_id = 0; 837 } 838 if (tbl_entry->pDOI_Dst_pr) { 839 *tbl_entry->pDOI_Dst_pr = 0; 840 } 841 842 tbl_entry->id1 = 0; 843 tbl_entry->id2 = 0; 844 RELEASE_TBL_ENTRY(pReasm->tblFree, tbl_entry); 845} 846 847/***************************************************************************** 848 Check if the datagram entry specified in tbl_entry in hash table is complete 849******************************************************************************/ 850static int _CheckDatagramComplete(reasmThread_t *pReasm, tbl_entry_t *tbl_entry, NtNetStreamRx_t hNetRx) 851{ 852 int size = 0, i; 853 int fstFrag = -1; 854 int lstFrag = -1; 855 856 /* Check all fragments in this datagram entry */ 857 for (i = 0; i < tbl_entry->fragCnt; i++) { 858 fstFrag = i; 859 if (tbl_entry->aFrag[i].lastFrag) { 860 lstFrag = i; 861 } 862 size += tbl_entry->aFrag[i].size; 863 } 864 865 /* if all received, process and release it */ 866 if (fstFrag >= 0 && lstFrag >= 0) { 867 /* Got both first and last frag */ 868 if (tbl_entry->aFrag[lstFrag].offset + tbl_entry->aFrag[lstFrag].size == size) { 869 pReasm->datagramCompleted++; 870 871 /* A complete datagram received and collected. Should do a checksum calculation on it to validate correctness */ 872 873 _ReleaseDgramTblEntry(pReasm, tbl_entry, hNetRx); 874 return 1; 875 } 876 } 877 return 0; 878} 879 880/***************************************************************************** 881 The main re-assembling thread routine. This function is called for each stream 882 specified in the load-balance configuration. It collects packets and fragments. 883 On fragment receival, it collects them into one datagram collection for 884 re-assembling. The hNetBufs are released. The thread may receive the fragments 885 from the stream it reads from, or it receives it from a msg box from one of 886 the un-matched threads. On receival of a first-fragment, the interest of that 887 datagram is raised in the DOI table associated with the un-matched thread 888 these fragments will be sent on. A first-fragment will always be sent on the 889 ordinary data streams, never on one of the un-matched streams specified. 890******************************************************************************/ 891#if defined(__linux__) || defined(__FreeBSD__) 892 static void *_ReassemblyThread(void *arg) 893#elif defined(WIN32) || defined (WIN64) 894 static unsigned __stdcall _ReassemblyThread(void *arg) 895#endif 896{ 897 reasmThread_t *pReasm = (reasmThread_t *)arg; 898 char tmpBuffer[20]; // Buffer used to create NTPL expressions 899 char errorBuffer[NT_ERRBUF_SIZE]; // Error buffer 900 NtNetStreamRx_t hNetRx; // Handle to the RX stream 901 NtNetBuf_t hNetBuf=NULL; // Net buffer container 902 int i,ii; // Loop counters 903 int status; // NTAPI call return value 904 uint8_t fromUnm; // Flag to indicate that the hNetBuf is sent from an un-matched thread 905 906 uint8_t unmIndex; 907 908 int timeoutChkCnt; // Counter to control packet timeout checking 909 uint64_t sysTs; // Variable to hold system time stamp 910 DOI_dgramId_u val, val1; // DOI datagram ID variables 911 tbl_entry_t *pTbl; // Hash table traverse helper variable 912 913 sprintf(tmpBuffer,"IPFReasm_%i", pReasm->streamId); 914 915 if((status = NT_NetRxOpen(&hNetRx, tmpBuffer, NT_NET_INTERFACE_PACKET, pReasm->streamId, -1)) != NT_SUCCESS) { 916 // Get the status code as text 917 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 918 fprintf(stderr, "[%i] NT_NetRxOpen() failed: %s\n", pReasm->streamId, errorBuffer); 919#if defined(__linux__) || defined(__FreeBSD__) 920 return (void *)NULL; 921#elif defined(WIN32) || defined (WIN64) 922 { _endthreadex(0); return 0; } 923#endif 924 } 925 926 timeoutChkCnt = 0; 927 while(pReasm->pIpDefrag->Running) { 928 929 hNetBuf=NULL; 930 /* Check all Msg boxes for fragment packets from un-matched threads */ 931 for (i = 0; i < pReasm->pIpDefrag->unmCnt; i++) { 932 if (!MSG_BOX_EMPTY(&pReasm->pMsgbox[i])) { 933 hNetBuf = MSG_BOX_GET(&pReasm->pMsgbox[i]); 934 fromUnm=1; 935 unmIndex = (uint8_t)i; 936 break; 937 } 938 } 939 940 if (hNetBuf==NULL) { 941 if((status = NT_NetRxGet(hNetRx, &hNetBuf, 1)) != NT_SUCCESS) { 942 if((status == NT_STATUS_TIMEOUT) || (status == NT_STATUS_TRYAGAIN)) { 943 944 if (pReasm->pIpDefrag->fragTimeout > 0) { 945 if (++timeoutChkCnt >= 100) { 946 int deleteIt = 0; 947 /* Run through list of fragments to delete timed out fragments */ 948 sysTs = GetSystemTimeNoArg()*100; // make it 1/10ns units 949 950 for (i = 0; i < REASSEMBLY_HASH_TBL_SIZE; i++) { 951 if (pReasm->tbl[i]) { 952 tbl_entry_t *pNxt, *tbl_entry = pReasm->tbl[i]; 953 while (tbl_entry) { 954 deleteIt = 0; 955 for (ii = 0; ii < tbl_entry->fragCnt; ii++) { 956 if (sysTs - NT_NET_GET_PKT_TIMESTAMP(tbl_entry->aFrag[ii].hNetBuf) > pReasm->pIpDefrag->fragTimeout) { 957 deleteIt=1; 958 break; 959 } 960 } 961 if (deleteIt) { 962 pNxt = tbl_entry->pNext; 963 _ReleaseDgramTblEntry(pReasm, tbl_entry, hNetRx); 964 pReasm->fragmentsTimedout++; 965 tbl_entry = pNxt; 966 } else { 967 tbl_entry = tbl_entry->pNext; 968 } 969 } 970 } 971 } 972 timeoutChkCnt = 0; 973 } 974 } 975 continue; 976 } 977 978 if (status) { 979 // Get the status code as text 980 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 981 fprintf(stderr, "[%i] NT_NetRxGet() failed: %s\n", pReasm->streamId, errorBuffer); 982 goto EXIT; 983 } 984 } 985 fromUnm = 0; 986 unmIndex = 0; 987 } 988 989 timeoutChkCnt = 0; 990 if (NT_NET_GET_PKT_L2_FRAME_TYPE(hNetBuf) == NT_L2_FRAME_TYPE_ETHER_II && 991 NT_NET_GET_PKT_L3_FRAME_TYPE(hNetBuf) == NT_L3_FRAME_TYPE_IPv4 && 992 NT_NET_GET_PKT_L3_FRAGMENTED(hNetBuf)) { 993 uint32_t unmEntry; 994 uint32_t src, dst; 995 uint8_t prot; 996 uint16_t id; 997 int offset; 998 tbl_entry_t *tbl_entry=NULL; 999 DOI_dgramTbl_t *unmTbl; 1000 1001 if (NT_NET_GET_PKT_L3_FIRST_FRAG(hNetBuf)) pReasm->firstFragRcv++; 1002 IPV4_GET_DGRAM_ID(hNetBuf, src, dst, id, prot); 1003 val.src = src; 1004 val.id = id; 1005 val1.src = dst; 1006 val1.id = prot; 1007 1008 LOOKUP_ENTRY(pReasm->tbl, tbl_entry_t, REASSEMBLY_HASH_TBL_SIZE, val.src_id, val1.src_id, tbl_entry); 1009 if (tbl_entry == NULL) { 1010 GET_NEW_TBL_ENTRY(pReasm->tblFree, tbl_entry_t, tbl_entry); 1011 } 1012 1013 if (tbl_entry->id1 == 0) { 1014 // New datagram fragment received 1015 tbl_entry->id1 = val.src_id; 1016 tbl_entry->id2 = val1.src_id; 1017 tbl_entry->fragCnt = 0; 1018 tbl_entry->pDOI_Src_id = NULL; 1019 tbl_entry->pDOI_Dst_pr = NULL; 1020 ADD_ENTRY_TO_TBL(pReasm->tbl, tbl_entry_t, REASSEMBLY_HASH_TBL_SIZE, tbl_entry); 1021 } else { 1022 // Error check 1023 if (tbl_entry->fragCnt == MAX_FRAG_CNT) { 1024 fprintf(stderr, "[%i] ERROR - MAX frag cnt reached\n", pReasm->streamId); 1025 exit(0); 1026 } 1027 } 1028 1029 // Add fragment to datagram in list 1030 tbl_entry->aFrag[tbl_entry->fragCnt].hNetBuf = hNetBuf; 1031 tbl_entry->aFrag[tbl_entry->fragCnt].fromUnm = fromUnm; 1032 tbl_entry->aFrag[tbl_entry->fragCnt].unmIndex = unmIndex; 1033 1034/* 1035* warning You may get conversion warnings here caused by ntohs(). This is a bug in /usr/include/bits/byteswap.h 1036*/ 1037 tbl_entry->aFrag[tbl_entry->fragCnt].offset = (uint16_t)IPV4_FRAGMENT_OFFSET(hNetBuf); 1038 tbl_entry->aFrag[tbl_entry->fragCnt].size = (uint16_t)IPV4_DATA_LEN(hNetBuf); 1039 tbl_entry->aFrag[tbl_entry->fragCnt].firstFrag = (uint8_t)NT_NET_GET_PKT_L3_FIRST_FRAG(hNetBuf); 1040 1041 if (pReasm->pIpDefrag->ExtDescrType == 9) { 1042 tbl_entry->aFrag[tbl_entry->fragCnt].lastFrag = (uint8_t)NT_NET_GET_PKT_IPF_LAST_FRAGMENT(hNetBuf); 1043 } else { 1044 tbl_entry->aFrag[tbl_entry->fragCnt].lastFrag = IPV4_LAST_FRAG(hNetBuf); 1045 } 1046 1047 tbl_entry->fragCnt++; 1048 1049 /* Check datagram for completion */ 1050 if (_CheckDatagramComplete(pReasm, tbl_entry, hNetRx) == 0) { 1051 if (NT_NET_GET_PKT_DESCRIPTOR_FORMAT(hNetBuf) == 9) { 1052 if (NT_NET_GET_PKT_L3_FIRST_FRAG(hNetBuf)) { 1053 int idx=-1; 1054 1055 /* Find thread-index of the un-matched stream receiving un-matched fragments */ 1056 offset = NT_NET_GET_PKT_IPF_UNMATCHED_STREAMID(hNetBuf) - pReasm->pIpDefrag->unmStart; 1057 /* Get a pointer to the DOI table between this re-assembling thread and the un-matching fragments thread */ 1058 unmTbl = &pReasm->pIpDefrag->pDOI_Tbls[(offset * pReasm->pIpDefrag->reasmCnt + pReasm->idx)* DOI_FRAG_TBL_SIZE]; 1059 DOI_HASH_GET_KEY(unmEntry, src, id); 1060 1061 // On first fragment receival - Notify the corresponding un-matched thread about interest in fragments of this datagram 1062 for (i=0;i<MAX_SRC_ID;i++) { 1063 if (unmTbl[unmEntry].aSrc_id[i] == val.src_id && 1064 unmTbl[unmEntry].aDst_pr[i] == val1.src_id ) { 1065 pReasm->Src_IdClash++; 1066 /* Ok already there! */ 1067 idx = i; 1068 break; 1069 } 1070 if (idx < 0 && unmTbl[unmEntry].aSrc_id[i] == 0) { 1071 idx = i; 1072 } 1073 } 1074 1075 if (idx >= 0) { 1076 tbl_entry->pDOI_Src_id = &unmTbl[unmEntry].aSrc_id[idx]; 1077 tbl_entry->pDOI_Dst_pr = &unmTbl[unmEntry].aDst_pr[idx]; 1078 *tbl_entry->pDOI_Dst_pr = val1.src_id; 1079 *tbl_entry->pDOI_Src_id = val.src_id; 1080 } else { 1081 fprintf(stderr, "[%i] Too many clashes in DOI (more than %i - raise MAX_SRC_ID)\n", pReasm->streamId, MAX_SRC_ID); 1082 exit(0); 1083 } 1084 } 1085 } 1086 } 1087 1088 } else { 1089 pReasm->nonFragments++; 1090 1091 // Release packet as it is not fragmented. 1092 if((status = NT_NetRxRelease(hNetRx, hNetBuf)) != NT_SUCCESS) { 1093 // Get the status code as text 1094 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1095 fprintf(stderr, "[%i] NT_NetRxRelease() failed: %s\n", pReasm->streamId , errorBuffer); 1096 goto EXIT; 1097 } 1098 } 1099 } 1100 1101EXIT: 1102 for (i = 0; i < REASSEMBLY_HASH_TBL_SIZE; i++) { 1103 pTbl = pReasm->tbl[i]; 1104 while (pTbl) { 1105 // Release all hash table fragment packets 1106 for (ii = 0; ii < pTbl->fragCnt; ii++) { 1107 if (pTbl->aFrag[ii].fromUnm) { 1108 _FreeUnmFragment(pReasm, pTbl->aFrag[ii].unmIndex, pTbl->aFrag[ii].hNetBuf); 1109 } else { 1110 NT_NetRxRelease(hNetRx, pTbl->aFrag[ii].hNetBuf); 1111 } 1112 } 1113 pTbl = pTbl->pNext; 1114 } 1115 } 1116 1117 /* Empty Msg boxes */ 1118 for (i = 0; i < pReasm->pIpDefrag->unmCnt; i++) { 1119 while (!MSG_BOX_EMPTY(&pReasm->pMsgbox[i])) { 1120 hNetBuf = MSG_BOX_GET(&pReasm->pMsgbox[i]); 1121 _FreeUnmFragment(pReasm, i, hNetBuf); 1122 pReasm->msgboxPackets++; 1123 } 1124 } 1125 1126 // Close the stream 1127 if ((status = NT_NetRxClose(hNetRx)) != NT_SUCCESS) { 1128 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1129 fprintf(stderr, "[%i] Re-asm thread: NT_NetRxGet() failed: %s\n", pReasm->streamId, errorBuffer); 1130 } 1131#if defined(__linux__) || defined(__FreeBSD__) 1132 return (void *)NULL; 1133#elif defined(WIN32) || defined (WIN64) 1134 { _endthreadex(0); return 0; } 1135#endif 1136} 1137 1138/***************************************************************************** 1139 Ctrl-C signal handler routine. 1140******************************************************************************/ 1141#if defined(WIN32) || defined (WIN64) 1142static BOOL WINAPI StopApplication(int sig) 1143#else 1144static void StopApplication(int sig __attribute__((unused))) 1145#endif 1146{ 1147#ifdef WIN32 1148 IpDefrag.Running = 0; 1149 return TRUE; 1150#else 1151 if (sig == SIGINT) 1152 IpDefrag.Running = 0; 1153#endif 1154} 1155 1156/* Default paramters */ 1157#define INPUT_ADAPTER 0 1158#define FRAGMENT_TIMEOUT 2000 // mSec 1159#define NUM_STREAMS 4 1160#define NUM_UNM_STREAMS 4 1161#define TABLE_TIMEOUT 125 1162#define TABLE_PERSIST_TIMEOUT 0 1163 1164/***************************************************************************** 1165 Main routine. It initializes, configures FPGA, starts needed threads 1166 (un-matched and re-assembling threads), and loops while printing 1167 out statistics. On Ctrl-C it stops looping and begins the cleanup process. 1168******************************************************************************/ 1169int main(int argc, const char *argv[]) 1170{ 1171 char tmpBuffer[100]; // Buffer to build filter string 1172 char errorBuffer[NT_ERRBUF_SIZE]; // Error buffer 1173 int status; // Status variable 1174 NtNtplInfo_t ntplInfo; // Return data structure from the NT_NTPL() call. 1175 NtInfoStream_t hInfoStream; // Info stream handle 1176 NtInfo_t hInfo; // Info handle 1177 int i, ii; // Counter variables 1178 int inp_1, inp_2; // Input port numbers for selected adapter 1179 NtConfigStream_t hCfgStream; // Handle to a config stream 1180 int hashTblFreed; // Counter for re-assemble hash table entries remained on exit 1181 int waitTblFreed; // Counter for un-matched wait table entries remained on exit 1182 int msgboxElmFreed; // Counter for hNetBuf elements remained in msg boxes 1183 int tablePersistTimeout; // Control parameter for NTPL IPFMode TablePersist setting 1184 int tableTimeout; // Control parameter for NTPL IPFMode timeout setting 1185 struct argparse argparse; 1186#define NUM_NTPL 3 1187 const char *ntplExpr[NUM_NTPL] = 1188 {"Assign[streamid=(%i..%i)]=port==(%i..%i)", 1189 "HashMode=Hash%iTuple", 1190 "IPFMode[StreamId=(%i..%i);timeout=%i;TablePersist=%s]=port==(%i..%i)"}; 1191 1192// Set up ctrl+c handler 1193#if defined(WIN32) || defined (WIN64) 1194 SetConsoleCtrlHandler((PHANDLER_ROUTINE)StopApplication, TRUE); 1195#else 1196 struct sigaction newaction; // Ctrl+c handle 1197 memset(&newaction, 0, sizeof(newaction)); 1198 newaction.sa_handler = StopApplication; 1199 if (sigaction(SIGINT, &newaction, NULL) < 0) { 1200 fprintf(stderr, "Failed to register sigaction.\n"); 1201 exit(1); 1202 } 1203#endif 1204 1205 // Setup default parameter settings 1206 memset(&IpDefrag, 0 , sizeof(struct _ipDefrag)); 1207 IpDefrag.adapterNo = INPUT_ADAPTER; 1208 IpDefrag.reasmCnt = NUM_STREAMS; 1209 IpDefrag.reasmStart = 0; 1210 IpDefrag.unmCnt = NUM_UNM_STREAMS; 1211 IpDefrag.unmStart = IpDefrag.reasmStart + IpDefrag.reasmCnt; 1212 IpDefrag.fragTimeout = FRAGMENT_TIMEOUT * 100000ULL; 1213 tableTimeout = TABLE_TIMEOUT * 10; 1214 tablePersistTimeout = TABLE_PERSIST_TIMEOUT; 1215 1216 argparse_init(&argparse, arg_options, usageText, 0); 1217 argparse_parse(&argparse, argc, argv); 1218 1219 if (opt_adapter != -1) { 1220 IpDefrag.adapterNo = opt_adapter; 1221 } 1222 if (opt_reasm != -1) { 1223 IpDefrag.reasmCnt = opt_reasm; 1224 } 1225 if (opt_unm != -1) { 1226 IpDefrag.unmCnt = opt_unm; 1227 } 1228 if (opt_frag != -1) { 1229 IpDefrag.fragTimeout = (uint64_t)opt_frag * 100000ULL; 1230 } 1231 if (opt_persist != NULL) { 1232 if (strcmp(opt_persist, "timeout") == 0) { 1233 tablePersistTimeout = 1; 1234 } 1235 } 1236 if (opt_timeout != -1) { 1237 tableTimeout = opt_timeout * 10; 1238 } 1239 1240 // Initialize the NTAPI library and thereby check if NTAPI_VERSION can be used together with this library 1241 if((status = NT_Init(NTAPI_VERSION)) != NT_SUCCESS) { 1242 // Get the status code as text 1243 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1244 fprintf(stderr, "NT_Init() failed: %s\n", errorBuffer); 1245 return -1; 1246 } 1247 1248 /* Open the info stream */ 1249 if ((status = NT_InfoOpen(&hInfoStream, "IPFExample")) != NT_SUCCESS) { 1250 // Get the status code as text 1251 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1252 fprintf(stderr, "NT_InfoOpen() failed: %s\n", errorBuffer); 1253 return -1; 1254 } 1255 1256 /* Read number of adapter */ 1257 hInfo.cmd=NT_INFO_CMD_READ_ADAPTER_V6; 1258 hInfo.u.adapter_v6.adapterNo = (uint8_t) IpDefrag.adapterNo; 1259 1260 if((status = NT_InfoRead(hInfoStream, &hInfo)) != 0) { 1261 // Get the status code as text 1262 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1263 fprintf(stderr, "NT_InfoRead() failed: %s\n", errorBuffer); 1264 return -1; 1265 } 1266 /* collect port range for the selected adapter */ 1267 inp_1 = hInfo.u.adapter_v6.data.portOffset; 1268 inp_2 = hInfo.u.adapter_v6.data.portOffset + hInfo.u.adapter_v6.data.numPorts - 1; 1269 1270 /* Ensure that Ext9 was configured. */ 1271 if (NT_PACKET_DESCRIPTOR_TYPE_NT_EXTENDED == hInfo.u.adapter_v6.data.descriptorType && 9 == hInfo.u.adapter_v6.data.extendedDescriptor) { 1272 IpDefrag.ExtDescrType = hInfo.u.adapter_v6.data.extendedDescriptor; 1273 } else { 1274 fprintf(stderr, "The packet descriptor is not Ext9. Please set 'PacketDescriptor = Ext9' in ntservice.ini file for the selected adapter.\n"); 1275 return -1; 1276 } 1277 1278 /* Get timesync configuration. Must use OS mode to run timeouts with this demo */ 1279 if (IpDefrag.fragTimeout) { 1280 hInfo.cmd=NT_INFO_CMD_READ_TIMESYNC_V4; 1281 hInfo.u.timeSync_v4.adapterNo = (uint8_t) IpDefrag.adapterNo; 1282 if ((status = NT_InfoRead(hInfoStream, &hInfo)) != NT_SUCCESS) { 1283 fprintf(stderr, "Failed to read timesync info.\n"); 1284 return -1; 1285 } 1286 1287 if (hInfo.u.timeSync_v4.data.timeRef != NT_TIMESYNC_REFERENCE_OSTIME) { 1288 fprintf(stderr, "The timesync reference clock on the selected adapter is not OS. Please set TimeSyncReferencePriority=OsTime in ntservice.ini file for the selected adapter\n"); 1289 return -1; 1290 } 1291 } 1292 1293 /* Close info stream */ 1294 if((status = NT_InfoClose(hInfoStream)) != NT_SUCCESS) { 1295 // Get the status code as text 1296 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1297 fprintf(stderr, "NT_InfoClose() failed: %s\n", errorBuffer); 1298 return -1; 1299 } 1300 1301 // Open a config stream to assign a filter to a stream ID. 1302 if((status = NT_ConfigOpen(&hCfgStream, "IPFExample")) != NT_SUCCESS) { 1303 // Get the status code as text 1304 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1305 fprintf(stderr, "NT_ConfigOpen() failed: %s\n", errorBuffer); 1306 return -1; 1307 } 1308 1309 /* Build final NTPL command strings */ 1310 for (i = 0; i < NUM_NTPL; i++) { 1311 switch (i) { 1312 case 0: sprintf(tmpBuffer, ntplExpr[i], IpDefrag.reasmStart, IpDefrag.reasmStart + IpDefrag.reasmCnt - 1, inp_1, inp_2); break; 1313 case 1: sprintf(tmpBuffer, ntplExpr[i], (IpDefrag.ExtDescrType == 9)?5:2); break; 1314 case 2: 1315 if (IpDefrag.ExtDescrType == 9) { 1316 if (tablePersistTimeout) { 1317 sprintf(tmpBuffer, ntplExpr[i], IpDefrag.unmStart, IpDefrag.unmStart + IpDefrag.unmCnt - 1, tableTimeout, "TimeoutOnly", inp_1, inp_2); 1318 } else { 1319 sprintf(tmpBuffer, ntplExpr[i], IpDefrag.unmStart, IpDefrag.unmStart + IpDefrag.unmCnt - 1, tableTimeout, "LastFragment", inp_1, inp_2); 1320 } 1321 } else { 1322 IpDefrag.unmCnt = 0; 1323 continue; 1324 } 1325 break; 1326 default: strcpy(tmpBuffer, ntplExpr[i]); break; 1327 } 1328 1329 // Assign NTPL expressions needed 1330 if((status = NT_NTPL(hCfgStream, tmpBuffer, &ntplInfo, NT_NTPL_PARSER_VALIDATE_NORMAL)) != NT_SUCCESS) { 1331 fprintf(stderr, "ERROR --> %s\n", tmpBuffer); 1332 // Get the status code as text 1333 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1334 fprintf(stderr, "NT_NTPL() failed: %s\n", errorBuffer); 1335 fprintf(stderr, ">>> NTPL errorcode: %X\n", ntplInfo.u.errorData.errCode); 1336 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[0]); 1337 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[1]); 1338 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[2]); 1339 return -1; 1340 } 1341 } 1342 1343 IpDefrag.Running = 1; 1344 IpDefrag.allReasmClosed = 0; 1345 1346 /* Create IPv4 Reassembly threads */ 1347 IpDefrag.pReasmThreads = calloc(1, sizeof(reasmThread_t) * IpDefrag.reasmCnt); 1348 if (IpDefrag.pReasmThreads == NULL) { 1349 fprintf(stderr, "Memory allocation failed\n"); 1350 return -1; 1351 } 1352 1353 /* Create communication message boxes. One msg box for each Rasm thread from each Unm thread */ 1354 /* Create twice - one for NetBuf send from unmatched thread to reassembling thread and one for */ 1355 /* returning the NetBuf again to the unmatched thread */ 1356 IpDefrag.pReasmMsgboxes = 1357 calloc(1, sizeof(msg_box_t) * IpDefrag.reasmCnt * 2 * 1358 /* IpDefrag.unmCnt == 0 when IpDefrag.ExtDescrType != 8 */ 1359 (IpDefrag.unmCnt == 0 ? 1 : IpDefrag.unmCnt)); 1360 1361 if (IpDefrag.pReasmMsgboxes == NULL) { 1362 fprintf(stderr, "Memory allocation failed\n"); 1363 return -1; 1364 } 1365 IpDefrag.pReasmReturnMsgboxes = (msg_box_t *)((char *)IpDefrag.pReasmMsgboxes + sizeof(msg_box_t) * IpDefrag.reasmCnt * IpDefrag.unmCnt); 1366 1367 /* initialize and start all IP fragment re-assembling threads */ 1368 for (i = 0; i < IpDefrag.reasmCnt; i++) { 1369 tbl_entry_t *tbl; 1370 1371 IpDefrag.pReasmThreads[i].pIpDefrag = &IpDefrag; 1372 IpDefrag.pReasmThreads[i].streamId = IpDefrag.reasmStart + i; 1373 IpDefrag.pReasmThreads[i].idx = i; 1374 IpDefrag.pReasmThreads[i].pMsgbox = &IpDefrag.pReasmMsgboxes[i * IpDefrag.unmCnt]; 1375 1376 /* Create initial hash table elements and fill in free list. May grow dynamically later */ 1377 IpDefrag.pReasmThreads[i].tblFree = calloc(1, sizeof(tbl_entry_t)); 1378 tbl = IpDefrag.pReasmThreads[i].tblFree; 1379 if (tbl) { 1380 for (ii = 1; ii < INITIAL_HASH_TBL_ENTRY_CNT; ii++) { 1381 tbl->pNext = calloc(1, sizeof(tbl_entry_t)); 1382 tbl=tbl->pNext; 1383 if (tbl == NULL) break; 1384 } 1385 } 1386 1387 if((status = pthread_create(&IpDefrag.pReasmThreads[i].thread, NULL, _ReassemblyThread, (void*)&IpDefrag.pReasmThreads[i])) != 0) { 1388 fprintf(stderr, "Unable to create Unmatched stream thread"); 1389 return status; 1390 } 1391 } 1392 1393 if (IpDefrag.unmCnt) { 1394 /* Create all DOI-tables for all un-matched threads */ 1395 IpDefrag.pDOI_Tbls = calloc(1, sizeof(DOI_dgramTbl_t) * DOI_FRAG_TBL_SIZE * IpDefrag.unmCnt * IpDefrag.reasmCnt); 1396 1397 /* Create Unmatched fragment threads */ 1398 IpDefrag.pUnmThreads = calloc(1, sizeof(unmThread_t) * IpDefrag.unmCnt); 1399 if (IpDefrag.pUnmThreads == NULL) { 1400 fprintf(stderr, "Memory allocation failed\n"); 1401 return -1; 1402 } 1403 1404 for (i = 0; i < IpDefrag.unmCnt; i++) { 1405 IpDefrag.pUnmThreads[i].pIpDefrag = &IpDefrag; 1406 IpDefrag.pUnmThreads[i].streamId = IpDefrag.unmStart + i; 1407 IpDefrag.pUnmThreads[i].pDOI = &IpDefrag.pDOI_Tbls[i * DOI_FRAG_TBL_SIZE * IpDefrag.reasmCnt]; 1408 IpDefrag.pUnmThreads[i].pMsgboxReturn = &IpDefrag.pReasmReturnMsgboxes[i * IpDefrag.reasmCnt]; 1409 1410 if((status = pthread_create(&IpDefrag.pUnmThreads[i].thread, NULL, _UnMatchedThread, (void*)&IpDefrag.pUnmThreads[i])) != 0) { 1411 fprintf(stderr, "Unable to create Unmatched stream thread"); 1412 return status; 1413 } 1414 } 1415 } 1416 1417 /* main statistics loop */ 1418 while (IpDefrag.Running) { 1419 int sum1,sum2,sum3,sum4,sum5; 1420 printf("Parameters:\n"); 1421 printf("Adapter number to run on %i\n", IpDefrag.adapterNo); 1422 printf("Number of streams/threads %i\n", IpDefrag.reasmCnt); 1423 printf("Number of un-matched streams/threads %i\n", IpDefrag.unmCnt); 1424 printf("Fragment timeout (ms) %i\n", (int)(IpDefrag.fragTimeout/100000)); 1425 printf("IPFMode table timeout (ms) %i\n", tableTimeout/10); 1426 printf("IPFMode TablePersist "); 1427 if (tablePersistTimeout) { 1428 printf("TimeoutOnly\n"); 1429 } else { 1430 printf("LastFragment\n"); 1431 } 1432 printf("\nRunning with extended descriptor %i\n", IpDefrag.ExtDescrType); 1433 printf("-------------------------------------------------------------------\n"); 1434 1435 printf("Re-assembling streams:\n"); 1436 printf(" # Completed First frag Deleted partial Non-fragmented Id\n"); 1437 printf(" (timed out) packets clashes\n"); 1438 1439 sum1 = sum2 = sum3 = sum4 = sum5 = 0; 1440 for (i = 0; i < IpDefrag.reasmCnt; i++) { 1441 printf("%2d:%8d, %8d, %10d, %14d, %12d\n", IpDefrag.pReasmThreads[i].streamId, 1442 IpDefrag.pReasmThreads[i].datagramCompleted, IpDefrag.pReasmThreads[i].firstFragRcv, 1443 IpDefrag.pReasmThreads[i].fragmentsTimedout, IpDefrag.pReasmThreads[i].nonFragments, 1444 IpDefrag.pReasmThreads[i].Src_IdClash); 1445 sum1 += IpDefrag.pReasmThreads[i].datagramCompleted; 1446 sum2 += IpDefrag.pReasmThreads[i].firstFragRcv; 1447 sum3 += IpDefrag.pReasmThreads[i].fragmentsTimedout; 1448 sum4 += IpDefrag.pReasmThreads[i].nonFragments; 1449 sum5 += IpDefrag.pReasmThreads[i].Src_IdClash; 1450 } 1451 printf("sum:%7d, %8d, %10d, %14d, %12d\n", sum1,sum2,sum3,sum4,sum5); 1452 1453 if (IpDefrag.unmCnt) { 1454 printf("Un-matched fragment streams:\n"); 1455 printf(" # Fragments Deleted frags\n"); 1456 printf(" received (timed out)\n"); 1457 for (i = 0; i< IpDefrag.unmCnt; i++) { 1458 printf("%2d:%8d,%12d\n", IpDefrag.pUnmThreads[i].streamId, IpDefrag.pUnmThreads[i].fragRecv, IpDefrag.pUnmThreads[i].fragsDeleted); 1459 } 1460 } 1461 printf("\n-------------------------------------------------------------------\n"); 1462 fflush(stdout); 1463 // Sleep 2 sec 1464 sleep(2); 1465 } 1466 1467 /* Cleanup section */ 1468 hashTblFreed = 0; 1469 waitTblFreed = 0; 1470 msgboxElmFreed = 0; 1471 for (i = 0; i < IpDefrag.reasmCnt; i++) { 1472 tbl_entry_t *tbl; 1473 pthread_join(IpDefrag.pReasmThreads[i].thread, NULL); 1474 /* Free all free table entries */ 1475 while (IpDefrag.pReasmThreads[i].tblFree) { 1476 tbl = IpDefrag.pReasmThreads[i].tblFree->pNext; 1477 free(IpDefrag.pReasmThreads[i].tblFree); 1478 IpDefrag.pReasmThreads[i].tblFree = tbl; 1479 } 1480 /* Free all hash table entries containing fragments */ 1481 for (ii = 0; ii < REASSEMBLY_HASH_TBL_SIZE; ii++) { 1482 while (IpDefrag.pReasmThreads[i].tbl[ii]) { 1483 tbl = IpDefrag.pReasmThreads[i].tbl[ii]->pNext; 1484 hashTblFreed++; // already released on thread termination 1485 free(IpDefrag.pReasmThreads[i].tbl[ii]); 1486 IpDefrag.pReasmThreads[i].tbl[ii] = tbl; 1487 } 1488 } 1489 msgboxElmFreed += IpDefrag.pReasmThreads[i].msgboxPackets; 1490 } 1491 free(IpDefrag.pReasmThreads); 1492 1493 IpDefrag.allReasmClosed = 1; 1494 1495 for (i = 0; i < IpDefrag.unmCnt; i++) { 1496 pthread_join(IpDefrag.pUnmThreads[i].thread, NULL); 1497 } 1498 1499 for (i = 0; i < IpDefrag.unmCnt; i++) { 1500 wait_list_t *pWait = IpDefrag.pUnmThreads[i].pWait; 1501 while (pWait) { 1502 waitTblFreed++; // already released on thread termination 1503 pWait=pWait->pNext; 1504 } 1505 } 1506 1507 free(IpDefrag.pUnmThreads); 1508 free(IpDefrag.pReasmMsgboxes); 1509 free(IpDefrag.pDOI_Tbls); 1510 1511 if (msgboxElmFreed || hashTblFreed || waitTblFreed) { 1512 printf("\n----------------------------------------------------------"); 1513 if (hashTblFreed) { 1514 printf("\n%i entries remained in hash table on exit", hashTblFreed); 1515 } 1516 if (waitTblFreed) { 1517 printf("\n%i entries remained in un-matched fragments wait-table on exit", waitTblFreed); 1518 } 1519 if (msgboxElmFreed) { 1520 printf("\n%i entries remained in msg boxes on exit", msgboxElmFreed); 1521 } 1522 printf("\n----------------------------------------------------------\n"); 1523 } 1524 1525 // Delete the filter 1526 snprintf(tmpBuffer, 20, "delete=%d", ntplInfo.ntplId); 1527 if((status = NT_NTPL(hCfgStream, tmpBuffer, &ntplInfo, NT_NTPL_PARSER_VALIDATE_NORMAL)) != NT_SUCCESS) { 1528 // Get the status code as text 1529 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1530 fprintf(stderr, "NT_NTPL() failed: %s\n", errorBuffer); 1531 fprintf(stderr, ">>> NTPL errorcode: %X\n", ntplInfo.u.errorData.errCode); 1532 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[0]); 1533 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[1]); 1534 fprintf(stderr, ">>> %s\n", ntplInfo.u.errorData.errBuffer[2]); 1535 return -1; 1536 } 1537 1538 // Close the config stream 1539 if((status = NT_ConfigClose(hCfgStream)) != NT_SUCCESS) { 1540 // Get the status code as text 1541 NT_ExplainError(status, errorBuffer, sizeof(errorBuffer)); 1542 fprintf(stderr, "NT_ConfigClose() failed: %s\n", errorBuffer); 1543 return -1; 1544 } 1545 return 0; 1546}