CN113839835B - An accurate monitoring system for Top-k flows based on small flow filtering - Google Patents

Abstract

The invention discloses a Top-k flow accurate monitoring system based on small flow filtration, which comprises: the small flow filter is used for filtering small flows in network flow, reducing storage resource expenditure caused by accurate storage of small flow information, improving flow size estimation precision and solving the problem of failure of the traditional filter on small flow filtration; the large flow monitoring table is used for precisely storing large flow information to track and count the number of large flow packets, and improves the Top-k flow identification precision; the large flow monitoring table comprises a list Ha Xiduo mapping algorithm and a probability replacement strategy; the Shan Haxi multi-mapping algorithm firstly calculates the fingerprint value of the stream according to the identifier of the stream, and then repeatedly selects part of bits from the fingerprint value to rearrange and combine so as to generate a plurality of hash values, thereby reducing the expenditure of hash calculation and enabling the Top-k stream to have enough candidate positions for selection and storage. The probability replacement strategy determines whether to evict the minimum stream by searching the minimum stream in the mapping bucket and generating a replacement probability according to the packet number of the minimum stream, thereby providing a storage location for a relatively larger stream. The invention filters small flow with small storage resource cost, then accurately monitors large flow to accurately count, has high space utilization rate, and can reach high Top-k flow identification rate with small space cost.

Description

An accurate monitoring system for Top-k flows based on small flow filtering

Technical field

The invention relates to the field of network measurement, and specifically to a high-precision Top-k flow identification system and a technical solution for filtering small flows in data flows.

Background technique

The tasks of network measurement include identification of Top-k flows, flow size estimation, flow quantity statistics, etc. It provides key information for analyzing the characteristics of network traffic and is the basis for network management and monitoring. Top-k flow identification is usually defined as finding the top k largest flows in network traffic, and the size of the flow is defined as the number of packets in the network data flow. Generally speaking, in order to identify the Top-k flows, network measurement programs will allocate a counter to the arriving data flow to track the size of the detected flow. However, for millions of network data flows, it is difficult to measure each piece of data. The stream maintains a counter. At the same time, in order to correctly identify the Top-k flows, the estimation error of the flow size by the measurement program needs to be guaranteed to be within a very small range. Therefore, under the premise of ensuring the processing speed of the algorithm, finding a high-precision and low-overhead Top-k flow identification method has become an important challenge for current Top-k flow identification methods.

At present, the main Top-k flow identification methods are roughly divided into three categories. The first type is a method based on sketch, and is divided into two structures: sketch and small top heap, and sketch and hash table. The sketch method of the first structure counts the sizes of all flows through a two-dimensional counter, and identifies the Top-k flows among them with the help of small top heap tracking. The sketch method of the second structure stores small flows through sketch, and the hash table monitors the large flows to reduce the cost of storage resources, and uses a replacement algorithm to expel small flows in the hash table to accurately store large flows. The second category is counter-based methods that accurately estimate the flow size of large flows by allocating a counter in the cache for them. The third category is a method based on filtering ideas. Filters are used to filter small flows in network traffic, and then large flows in network traffic are extracted to avoid the impact of small flows on the accurate counting of large flows and improve the estimation accuracy of flow size.

But you will also face the following problems:

1. Unable to meet the requirements of high precision and low memory overhead at the same time

As the number of network devices on the Internet continues to increase, the number of network data flows has already reached millions, and the size of network traffic obeys a heavy-tail distribution, that is, a small number of large flows in the network occupy most of the data in the network traffic. packets, while the large number of small flows only accounts for a small number of data packets in network traffic. In this regard, the sketch-based method must use a sufficient number of counters to reduce hash collisions, and each counter must use enough bits to avoid overflow, thus failing to reduce the cost of storage resources. The counter-based method must also allocate enough counters to track a huge number of flows, and there is a problem of misjudgment of a small flow as a large flow, which affects the accuracy of Top-k flow identification.

2. Transitional filtration and filter failure problems of traditional small flow filters

Traditional filters use a two-dimensional counter array to record the number of packets arriving in a flow, and when the values of all counters mapped to the data flow reach the threshold T, they will be allowed to pass the filter. However, most of the counters in the filter will reach the threshold T after a period of time, causing all flows to pass through the filter directly, making the filter unable to filter small flows. Although existing filters measure a fixed time as a period to reset the counter in the filter, this avoids the counter in the filter always remaining in the state of reaching the threshold. However, after the filter counter is reset, the large flow needs to re-increment the counter value in the filter in order to pass the filter, causing the data packets of the large flow to be accidentally consumed, thus causing the number of data packets of the large flow to be underestimated. That is, the transition filtering problem.

Comparative document: CN111262756A discloses an accurate measurement method for high-speed network elephant flows. It first filters the small flows in the data flow through a sketch-based filter, and then extracts the large flows in the network traffic through an extractor based on cuckoo hashing. Reduce storage resource overhead for small flows and accurately track large flows to improve the recognition rate of large flows. The comparison file solution does not solve the problem of filtering failure, which makes the small flow filter unable to filter small flows in network traffic after working for a period of time. The extraction method based on cuckoo hash has the problem of low accuracy in identifying large flows. .

Contents of the invention

The technical problem to be solved by this invention is to propose a small flow filter data structure based on sketch technology, using a combination of dual counters, and using a method of periodically updating the counters in the filter, and designing strategies corresponding to two counter updates respectively. Accurately record the arrival of each flow in each cycle to identify large flows and small flows in network data flows. At the same time, a large flow monitoring table is designed based on a single hash multi-map hash algorithm to ensure that Top-k flows have sufficient location storage, and a probabilistic replacement strategy is used to expel the smallest flow, accurately store large flows, and improve Recognition accuracy of Top-k flows.

In order to solve the above technical problems, the present invention adopts the following technical solutions:

The present invention provides a Top-k flow accurate monitoring system based on small flow filtering, including:

The small flow filter is used to distinguish large flows and small flows in network traffic, so as to extract the large flows for accurate tracking of the number of statistic packets; the small flow filter uses two small counters in pairs to record the flow Different packet number information to achieve low memory space overhead, and update the counter in itself periodically; the two small counters are used to record the average number of packets arriving in each cycle and the number of packets arriving in the current cycle. ;

The large flow monitoring table is used to accurately monitor large flows in network traffic and accurately count the number of packets in large flows; the large flow monitoring table is a hash table composed of hash buckets, and each hash bucket can store multiple For each stream, a single hash multi-mapping algorithm is used to map each stream into multiple candidate hash buckets to ensure that the Top-k flows have sufficient location storage, and a probabilistic replacement strategy is adopted to accurately monitor large flows; the single hash bucket The hash multi-mapping algorithm generates multiple hash values through one hash calculation to map to multiple hash buckets; the probability replacement strategy is to use a certain probability to replace all candidate positions when there are no vacancies. Minimum flow among candidate locations;

The small flow filter is composed of d arrays, each array is composed of w buckets, and each bucket contains two counters, namely a new counter and an old counter; the new counter records the packets arriving in the current cycle. Number; the old counter records the average number of packets arriving in the flow in the past cycle;

The large flow monitoring table is composed of r hash buckets, each hash bucket contains c slots, and each slot stores the fingerprint value FP and the packet number counter of a flow, that is, each slot stores a flow.

The method of the present invention also provides technical solutions based on the above system, including:

When the data packet arrives, the small flow filter first maps the flow identifier to a certain bucket in the d array through d two independent hash functions, and obtains the smallest new counter value and the smallest value in the d buckets. The old counter value is used, and the minimum new counter value is used as the number of packets arriving in the current flow, and the minimum old counter value is used as the average number of packets arriving in the flow in the past cycle. When the minimum new counter value of a flow reaches the threshold T, the flow is considered a newly arrived large flow and is allowed to pass the filter and enter the large flow monitoring table. When the minimum old counter value of a flow reaches the threshold T, the flow is considered to be a continuously arriving large flow and is allowed to pass the filter and enter the large flow monitoring table.

The two counters are more concerned about whether the threshold T is reached, and the threshold T is usually very small. Therefore, the two counters only need to be set to a few bit sizes to achieve the purpose of small size and low overhead.

When the data packet of the flow arrives, the large flow monitoring table first calculates the fingerprint value FP through the hash function based on the flow identifier, and then randomly selects a fixed number of bits from it multiple times and arranges it to generate multiple sub-hash values. Thus mapped to multiple hash buckets. Subsequently, the large flow monitoring table checks all mapping buckets. If the flow has been stored, the counter corresponding to the flow is increased by 1; if the flow is not stored but there is a gap, the flow is inserted into a gap; if the flow is not stored, If there are no vacancies, the minimum flow in the mapping bucket is found, and a replacement probability is generated based on the number of packets in the minimum flow to decide whether to replace the minimum flow with the newly arrived flow.

Further, the Top-k flow identification system includes the following operations:

1. Insertion and reporting of small flow filters;

The small flow filter maps each arriving packet to a bucket on each counter array, reports whether the packet can pass the filter based on the minimum new counter value and the minimum old counter value, and decides whether to update the new counter in it .

2. Periodic update of the counter in the small flow filter;

When a small flow filter measures a certain number of packets, it will update all counters in itself. The new counter will be directly reset to 0, and the old counter will adopt a half update strategy, that is, it will be updated to the average value of the new counter value and the old counter value in the previous cycle.

3. Insertion of large flow monitoring table;

When a data packet is introduced into the large flow monitoring table, the fingerprint value of the outgoing flow is first counted based on the flow identifier of the flow to which the data packet belongs, and then the flow fingerprint value is queried in the monitoring table, and the query result and hash are Different update steps are performed depending on whether there is an empty space in the bucket.

4. Replacement of large flow monitoring table;

When a data packet of a flow arrives, all its mapped hash buckets are full, and the flow is not recorded in the bucket, a replacement probability 1/(C _min is first generated based on the number of packets of the smallest flow in the bucket C _min +1) and then compare it with a randomly generated real number between 0 and 1. If the replacement probability is greater than a real number, the smallest flow is replaced, otherwise, the packets of this flow will be discarded.

5. Top-k flow report of large flow monitoring table;

The large flow monitoring table first sorts all the flows from large to small according to the number of packets in the flow, extracts the top k flows, adds the threshold T of the small flow filter as the final number of packets of the flow, and then reports these k flows to the server. The streams are Top-k streams.

The beneficial effects of the present invention are:

1. The present invention uses two small counters to construct a small flow filter, and periodically updates the counters in the small flow filter. One counter records the number of packets that a flow has arrived in the current cycle to identify newly arriving large flows, and the other counter records the average number of packets that a flow has arrived in the past cycle to identify large flows that continue to arrive. The two small counters are combined to identify large flows in network traffic, reduce the waste of storage space caused by storing small flows, solve the problem of traditional filters failing to filter small flows, and improve the identification accuracy of Top-k flows.

2. The present invention combines a single hash multi-mapping algorithm to design a low-overhead and high-precision Top-k flow identification method. First, the fingerprint value is calculated based on the flow identifier, and then bits are selected from the fingerprint value to reassemble a hash value to be mapped to the hash table, reducing the cost of hash calculation. At the same time, since each stream can have multiple candidate hash buckets, it can be ensured that the Top-k stream has enough storage locations for it to choose from, avoiding the problem that Top-k cannot monitor its stream size due to no location storage. , improve the Top-k flow recognition accuracy.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a structural diagram of the Top-k flow accurate monitoring system based on small flow filtering according to the method of the present invention.

Figure 2 is a data structure diagram of the small flow filter in the method of the present invention.

Figure 3 is a data structure diagram of the large flow monitoring table in the method of the present invention.

Figure 4 is a flow chart of inserting and releasing data packets by the small flow filter in the method of the present invention.

Figure 5 is a flow chart of periodic updating of the counter in the small flow filter in the method of the present invention.

Figure 6 is a flow chart of large flow monitoring table data packet insertion in the method of the present invention.

Figure 7 is a flow chart of flow replacement in the large flow monitoring table in the method of the present invention.

Figure 8 is a flow chart of the large flow monitoring table reporting Top-k flows in the method of the present invention.

Detailed ways

In order to better explain the content of the invention, the invention will be further verified below through specific examples. It is hereby explained that the embodiments are only for describing the present invention more directly. They are only a part of the present invention and cannot constitute any limitation to the present invention.

As shown in Figure 1, an embodiment of the present invention provides a Top-k flow accurate monitoring system based on small flow filtering, including:

The small flow filter is used to distinguish large flows and small flows in network traffic, so as to extract the large flows for accurate tracking of the number of statistic packets; the small flow filter uses two small counters in pairs to record the flow Different packet number information to achieve low memory space overhead, and update the counter in itself periodically; the two small counters are used to record the average number of packets arriving in each cycle and the number of packets arriving in the current cycle. ;

As shown in Figure 2, the small flow filter consists of d arrays, each array consists of w buckets, each bucket contains a pair of counters, namely a new counter and an old counter; the sizes of the two small counters are 4 bits.

The large flow monitoring table is used to accurately monitor large flows in network traffic and accurately count the number of packets in large flows; the large flow monitoring table is a hash table composed of hash buckets, and each hash bucket can store multiple For each stream, a single hash multi-mapping algorithm is used to map each stream into multiple candidate hash buckets to ensure that the Top-k flows have sufficient location storage, and a probabilistic replacement strategy is adopted to accurately monitor large flows; the single hash bucket The hash multi-mapping algorithm generates multiple hash values through one hash calculation to map to multiple hash buckets; the probability replacement strategy is to use a certain probability to replace all candidate positions when there are no vacancies. Minimum flow among candidate locations;

As shown in Figure 3, the large flow monitoring table consists of r hash buckets. Each hash bucket contains c slots. Each slot stores the fingerprint value FP and packet number counter of a flow, that is, each slot stores a flow. When the packet P _fid arrives, we calculate the flow fingerprint FP through the hash function H(.) and map it to i buckets through the sub-hash function subH _i (). The calculation of the hash value of the sub-hash function subH _i (.) is divided into two steps: (1) Select n bits at a fixed position from the fingerprint value FP value, and then the bits at the corresponding position will always be selected; (2) ) arranges the selected n bit values to generate a new hash value.

This embodiment also provides a technical solution based on the above system, including the following steps:

When a data packet arrives, the small flow filter first maps the flow identifier to a bucket in d arrays through d two independent hash functions, and obtains the smallest new counter value and the smallest old counter in the d buckets. The minimum new counter value is used as the number of packets arriving in the current flow, and the minimum old counter value is used as the average number of packets arriving in the flow in the past cycle. When the minimum new counter value of a flow reaches the threshold T, the flow is considered a newly arrived large flow and is allowed to pass through the filter. When the minimum old counter value of a flow reaches the threshold T, the flow is considered to be a continuously arriving large flow and is allowed to pass through the filter.

When the data packet of the flow arrives, the large flow monitoring table first calculates the fingerprint value FP through the hash function based on the flow identifier, and then randomly selects a fixed number of bits from it multiple times and arranges it to generate multiple sub-hash values, thereby mapping into multiple hash buckets. Subsequently, the large flow monitoring table checks all mapping buckets. If the flow has been stored, the counter corresponding to the flow is increased by 1; if the flow is not stored but there is a gap, the flow is inserted into a gap; if the flow is not stored, If there are no vacancies, the minimum flow in the mapping bucket is found, and a replacement probability is generated based on the number of packets in the minimum flow to decide whether to replace the minimum flow with the newly arrived flow.

1. The small flow filter inserts and releases data packets;

As shown in Figure 4, the small flow filter maps each arriving packet to a bucket on each counter array, reports whether the packet can pass the filter based on the minimum new counter value and the minimum old counter value, and decides Whether to update new counters in it.

First analyze the packet header information and extract the flow identifier; then map it to a bucket on the d arrays of the small flow filter through d hash functions, obtain the minimum new counter value and the minimum old counter value, and then compare the minimum values with threshold T. When the minimum new value is greater than the threshold, data packets are allowed to pass through the filter and enter the large flow monitoring table. Otherwise, update the new values in d mapping buckets and determine whether the minimum old value is greater than the threshold. If the minimum old value is greater than the threshold, the data packet is allowed to pass through the filter and enter the large flow monitoring table.

2. Periodic update of the counter in the small flow filter;

As shown in Figure 5, when the small flow filter measures a certain number of packets, it will update all counters in itself. The filter will update the counters in the buckets starting from the first bucket of each array until the last bucket is updated.

The new counter will be directly reset to 0, and the old counter will adopt a half update strategy, that is, it will be updated to the average value of the new counter value and the old counter value in the previous cycle.

3. Insertion of large flow monitoring table;

As shown in Figure 6, when a data packet is introduced into the large flow monitoring table, the fingerprint value of the outgoing flow is first counted based on the flow identifier of the flow to which the data packet belongs, and then the monitoring table is queried based on the fingerprint value of the flow, and Different update steps are performed based on the query results and whether there are vacancies in the hash bucket.

First, the fingerprint value FP is generated based on the flow identifier fid, and then the positions of k hash buckets are obtained through a single hash multi-mapping algorithm, and the flows in the buckets are queried sequentially. When the first gap is encountered, the stream is inserted and terminated. Or when the stream is queried, add 1 to the counter of the stream and end it. Otherwise, a new stream item to be replaced will be created and the stream replacement operation will be entered.

4. Replacement of large flow monitoring table;

As shown in Figure 7, when a data packet of a flow arrives, all its mapped hash buckets are full, and the flow is not recorded in the bucket, a replacement is first generated based on the number of packets C _min of the smallest flow in the bucket. Probability 1/(C _min +1), and then compare it with a randomly generated real number between 0 and 1. If the replacement probability is greater than a real number, the smallest flow is replaced, otherwise, the packets of this flow will be discarded.

5. Top-k flow report of large flow monitoring table;

As shown in 8, the large flow monitoring table first arranges all the flows from large to small according to the number of packets of the flow, extracts the top k flows, and adds the threshold T of the small flow filter as the final number of packets of the flow, and then Report these k flows to the server as Top-k flows.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the technical field of the present invention may make various modifications or additions to the described specific embodiments or substitute them in similar ways, but this will not deviate from the spirit of the present invention or exceed the definition of the appended claims. range.

Claims (6)

1. A Top-k flow accurate monitoring system based on small flow filtering, which is characterized by: The small flow filter is used to distinguish large flows and small flows in network traffic, so as to extract the large flows for accurate tracking of the number of statistic packets; the small flow filter uses two small counters in pairs to record the flow Different packet number information to achieve low memory space overhead, and update the counter in itself periodically; the two small counters are used to record the average number of packets arriving in each cycle and the number of packets arriving in the current cycle. ; The large flow monitoring table is used to accurately monitor large flows in network traffic and accurately count the number of packets in large flows; the large flow monitoring table is a hash table composed of hash buckets, and each hash bucket can store multiple For each stream, a single hash multi-mapping algorithm is used to map each stream into multiple candidate hash buckets to ensure that the Top-k flows have sufficient location storage, and a probabilistic replacement strategy is adopted to accurately monitor large flows; the single hash bucket The hash multi-mapping algorithm generates multiple hash values through one hash calculation to map to multiple hash buckets; the probability replacement strategy is to use a certain probability to replace all candidate positions when there are no vacancies. Minimum flow among candidate locations; The small flow filter is composed of d arrays, each array is composed of w buckets, and each bucket contains a pair of counters, namely a new counter and an old counter; the new counter records the packets arriving in the current cycle. Number; the old counter records the average number of packets arriving in the flow in the past cycle; The large flow monitoring table is composed of r hash buckets, each hash bucket contains c slots, and each slot stores the fingerprint value FP and the packet number counter of a flow, that is, each slot stores a flow. 2. A method based on the system of claim 1, characterized in that it includes the following steps: When the data packet arrives, the small flow filter first maps the flow identifier to a certain bucket in the d array through d two independent hash functions, and obtains the smallest new counter value and the smallest value in the d buckets. The old counter value, and the minimum new counter value is used as the number of packets arriving in the current flow, and the minimum old counter value is used as the average number of packets arriving in the flow in the past cycle; when the minimum new counter value of the flow reaches the threshold T, it is considered This flow is a newly arrived large flow, which is allowed to pass through the filter and enter the large flow monitoring table; when the minimum old counter value of the flow reaches the threshold T, the flow is considered to be a continuously arriving large flow, and is allowed to pass through the filter and enter the large flow monitoring table. Enter the large flow monitoring table; When the data packet of the flow arrives, the large flow monitoring table first calculates the fingerprint value FP through the hash function based on the flow identifier, and then randomly selects a fixed number of bits from it multiple times and arranges it to generate multiple sub-hash values. Thus, it is mapped to multiple hash buckets; then, the large flow monitoring table checks all mapping buckets. If the flow has been stored, the counter corresponding to the flow is incremented by 1; if the flow is not stored but there is a vacancy, the flow is added to the counter. A gap is inserted into the flow; if the flow is not stored and there is no gap, the minimum flow in the mapping bucket is found, and a replacement probability is generated based on the number of packets in the minimum flow to decide whether to replace the minimum flow with the newly arrived flow. 3. The method according to claim 2, characterized in that the Top-k flow accurate monitoring system consists of two modules: a small flow filter and a large flow monitoring table, specifically including the following operations: where a and b belong to small flow. Flow filters, c, d, and e belong to the large flow monitoring table module; a. Insertion and reporting of small flow filters; The small flow filter maps each arriving packet to a bucket on each counter array, reports whether the packet can pass the filter based on the minimum new counter value and the minimum old counter value, and decides whether to update the new counter in it ; b. Periodic update of the counter in the small flow filter; When the small flow filter measures a certain number of data packets, it will update all counters in itself; the new counters will be directly reset to 0, and the old counters will adopt a half update strategy, that is, they will be updated to the new ones in the previous cycle. The average of the counter value and the old counter value; c. Insertion of large flow monitoring table; When a data packet is introduced into the large flow monitoring table, the fingerprint value of the outgoing flow is first counted based on the flow identifier of the flow to which the data packet belongs, and then the flow fingerprint value is queried in the monitoring table, and the query result and hash are Different update steps are performed depending on whether there is an empty space in the bucket; d. Replacement of large flow monitoring table; When a data packet of a flow arrives, all its mapped hash buckets are full, and the flow is not recorded in the bucket, a replacement probability 1/(C _min is first generated based on the number of packets of the smallest flow in the bucket C _min +1), and then compare it with a randomly generated real number between 0 and 1; if the replacement probability is greater than the real number, replace the minimum flow, otherwise, the data packet of this flow will be discarded; e. Top-k flow report of large flow monitoring table; The large flow monitoring table first sorts all the flows from large to small according to the number of packets in the flow, extracts the top k flows, adds the threshold T of the small flow filter as the final number of packets of the flow, and then reports these k flows to the server. The streams are Top-k streams. 4. The method according to claim 3, characterized in that two small counters are used to construct a small flow filter, and the counters in the small flow filter are updated periodically; one of the counters records the arrival of the flow in the current period. The number of packets to identify newly arriving large flows, and another counter to record the average number of packets arriving in the flow in the past cycle to identify large flows that continue to arrive; the two small counters are combined to identify large flows in network traffic. Streams, reducing the waste of storage space caused by storing small flows, solving the problem of traditional filters failing to filter small flows, and improving the recognition accuracy of Top-k flows. 5. The method according to claim 3, characterized in that a low-overhead and high-precision Top-k flow identification method is designed in combination with a single hash multi-mapping algorithm; first, the fingerprint value calculated based on the flow identifier, Then select bits from the fingerprint value to reconstruct a hash value to map to the hash table, reducing the cost of hash calculation. 6. The method according to claim 3, characterized in that since each flow can have multiple candidate hash buckets, it can be ensured that the Top-k flow has enough storage locations for it to choose, avoiding Top-k The problem of being unable to monitor the flow size due to the lack of location storage improves the Top-k flow identification accuracy; at the same time, when replacing the minimum flow, since the minimum flow can be selected from multiple candidate positions, the minimum flow eviction can be accurately selected.