Download to read offline
![BUCKETINGBUCKETING
number: 13,200,393
hash: 2,005,620,294
bits: [100010110101011001000110]
100010110101011001 000110
---------------|------
value index](https://image.slidesharecdn.com/slides-190918035040/75/An-introduction-to-probabilistic-data-structures-69-2048.jpg)
![BUCKETINGBUCKETING
number: 13,200,393
hash: 2,005,620,294
bits: [100010110101011001000110]
value: number of zeros +1 (rtl)
100010110101011001 000110
---------------|------
value index](https://image.slidesharecdn.com/slides-190918035040/75/An-introduction-to-probabilistic-data-structures-70-2048.jpg)
![BUCKETINGBUCKETING
number: 13,200,393
hash: 2,005,620,294
bits: [100010110101011001000110]
value: number of zeros +1 (rtl)
index: The lowest b bits used to determine the
index of the register whose value is to be updated.
(m=2b)
100010110101011001 000110
---------------|------
value index](https://image.slidesharecdn.com/slides-190918035040/75/An-introduction-to-probabilistic-data-structures-71-2048.jpg)
![BUCKETINGBUCKETING
number: 13,200,393
hash: 2,005,620,294
bits: [100010110101011001000110]
value: number of zeros +1 (rtl)
index: The lowest b bits used to determine the
index of the register whose value is to be updated.
(m=2b)
each bucket will serve as an "estimator"
100010110101011001 000110
---------------|------
value index](https://image.slidesharecdn.com/slides-190918035040/75/An-introduction-to-probabilistic-data-structures-72-2048.jpg)
An introduction to probabilistic data structures
- 1.
- 2. INTROINTRO What is aProbabilistic Data Structure
- 3.
- 4. TRADEOFFSTRADEOFFS Trade accuracy forspeed meaning: they do not give 100% accurate results
- 5. TRADEOFFSTRADEOFFS Trade accuracy forspeed meaning: they do not give 100% accurate results Have less space requirements, also called sublinear
- 6. TRADEOFFSTRADEOFFS Trade accuracy forspeed meaning: they do not give 100% accurate results Have less space requirements, also called sublinear meaning: to count N distinct items, the required space is less than N
- 7. TRADEOFFSTRADEOFFS Trade accuracy forspeed meaning: they do not give 100% accurate results Have less space requirements, also called sublinear meaning: to count N distinct items, the required space is less than N Bonus: are associative
- 8.
- 9. GOOD ENOUGHGOOD ENOUGH Aquery may return a wrong answer But the answer is good enough (ex: count=1355, real count = 1299)
- 10. GOOD ENOUGHGOOD ENOUGH Aquery may return a wrong answer But the answer is good enough (ex: count=1355, real count = 1299) Usually for BigData(tm) whatever that is
- 11.
- 13. WHATS IT FOR?WHATSIT FOR? Membership tests Does this SET contains a particular ELEMENT?
- 14. An element iseither MAYBE on the set, or IS NOT
- 15. FALSE POSITIVES AREPOSSIBLEFALSE POSITIVES ARE POSSIBLE An element is either MAYBE on the set, or IS NOT
- 16. FALSE POSITIVES AREPOSSIBLEFALSE POSITIVES ARE POSSIBLE FALSE NEGATIVES NEVER HAPPENFALSE NEGATIVES NEVER HAPPEN An element is either MAYBE on the set, or IS NOT
- 17. HOW DOES ITWORK?HOW DOES IT WORK? IT'S A BIT SETIT'S A BIT SET
- 18. ADDINGADDING Note that avery long string still occupies the same couple of bits.
- 19.
- 20.
- 21.
- 22. PARAMETERSPARAMETERS Bit eld size(m) Number of hash functions (k) insertion and membership are O(k)
- 23.
- 24. RULESRULES One byte peritem in the input set gives about a 2% false positive rate 1024 elements to a 1KB Bloom Filter, about a 2% false positives
- 25. RULESRULES One byte peritem in the input set gives about a 2% false positive rate 1024 elements to a 1KB Bloom Filter, about a 2% false positives The optimal number of hash functions is about 0.7 times the number of bits per item 3 at a 10% false positive rate 13 at a 0.01% false positive rate
- 26. RULESRULES One byte peritem in the input set gives about a 2% false positive rate 1024 elements to a 1KB Bloom Filter, about a 2% false positives The optimal number of hash functions is about 0.7 times the number of bits per item 3 at a 10% false positive rate 13 at a 0.01% false positive rate The number of hashes dominates performance
- 27.
- 28. DIATRIBE: HASHINGDIATRIBE: HASHING JUSTUSE MURMUR3JUST USE MURMUR3
- 29. Fraction of keyshashed without collision (64 bits)
- 30. V{n,i} = Numberof items of namespace n hashed to the i-th bin Variance vs Mean for random distribution
- 32. BLACK=50% FLIP-PROBABILITY,BRIGHT GREEN=OUTPUTBIT IS “STUCK”- DOESN’T EVERBLACK=50% FLIP-PROBABILITY,BRIGHT GREEN=OUTPUT BIT IS “STUCK”- DOESN’T EVER VARYVARY
- 34. COUNT MIN SKETCHCOUNTMIN SKETCH
- 35. WHATS IT FOR?WHATSIT FOR? Top-K frequencies/Heavy hitters
- 36. WHATS IT FOR?WHATSIT FOR? Top-K frequencies/Heavy hitters How many times have you seen X? Leaderboards Stats Rate limiting, packet stats, etc
- 37. HOW DOES ITWORK?HOW DOES IT WORK? IT'S A 2D ARRAYIT'S A 2D ARRAY
- 38.
- 39.
- 40. PARAMETERSPARAMETERS Number of hashfunctions Size of matrix
- 41.
- 42. WHATS IT FOR?WHATSIT FOR? Quantiles
- 43. WHATS IT FOR?WHATSIT FOR? Quantiles What's the 90% percentile for GET /my/service? and 99%?
- 44. WHATS IT FOR?WHATSIT FOR? Quantiles What's the 90% percentile for GET /my/service? and 99%? anomaly detection: trigger at some percentile threshold
- 45. WHATS IT FOR?WHATSIT FOR? Quantiles What's the 90% percentile for GET /my/service? and 99%? anomaly detection: trigger at some percentile threshold quantiles per metric per user/location/etc
- 46. WHATS IT FOR?WHATSIT FOR? Quantiles What's the 90% percentile for GET /my/service? and 99%? anomaly detection: trigger at some percentile threshold quantiles per metric per user/location/etc Normally you need the full data set for a given quantile You cannot calculate a quantile of quantiles - makes it hard to do streaming
- 47. HOW DOES ITWORK?HOW DOES IT WORK? SPARSEREPRESENTATION OFTHECUMULATIVEDISTRIBUTION FUNCTIONSPARSEREPRESENTATION OFTHECUMULATIVEDISTRIBUTION FUNCTION
- 48. HOW DOES ITWORK?HOW DOES IT WORK? SPARSEREPRESENTATION OFTHECUMULATIVEDISTRIBUTION FUNCTIONSPARSEREPRESENTATION OFTHECUMULATIVEDISTRIBUTION FUNCTION After ingesting data, the data structure has learned the "interesting" points of the CDF, called centroids
- 49. HOW DOES ITWORK?HOW DOES IT WORK? SOMEDATASOMEDATA
- 50. HOW DOES ITWORK?HOW DOES IT WORK? EMPIRICAL CDFEMPIRICAL CDF
- 51. HOW DOES ITWORK?HOW DOES IT WORK? "INTERESTING"POINTS"INTERESTING"POINTS
- 52. COMBININGCOMBINING Create a newt-Digest and treat the internal centroids of the two left-hand side digests as incoming data The resulting t-Digest is a only slightly larger, but more accurate tDigest1 + tDigest2 = tDigest3 ------------------- -------- incoming data => new tDigest
- 53. QUERYINGQUERYING 8mb of pareto-distributeddata into a t-Digest
- 54. QUERYINGQUERYING 8mb of pareto-distributeddata into a t-Digest Resulting size was 5kb any percentile or quantile desired accuracy was on the order of 0.002%.
- 55. PARAMETERSPARAMETERS Compression tradeoff of sizevs accuracy depends on the implementation, some expose more params than others doesn't always mean the same thing
- 56.
- 58. WHATS IT FOR?WHATSIT FOR? Cardinality Estimation
- 59. WHATS IT FOR?WHATSIT FOR? Cardinality Estimation How many distinct ITEMS are there today? and yesterday? and the two days? ex: unique visitors
- 60. WHATS IT FOR?WHATSIT FOR? Cardinality Estimation How many distinct ITEMS are there today? and yesterday? and the two days? ex: unique visitors group-by/count without keeping all the data
- 61. HOW DOES ITWORK?HOW DOES IT WORK? IT'S COMPLICATEDIT'S COMPLICATED ...The observation that the cardinality of a multiset of uniformly-distributed random numbers can be estimated by calculating the maximum number of leading zeros in the binary representation of each number in the set. If the maximum number of leading zeros observed is n, an estimate for the number of distinct elements in the set is 2^n.
- 64. REAL SLOWNOWREAL SLOWNOW Ina random stream of integers
- 65. REAL SLOWNOWREAL SLOWNOW Ina random stream of integers ~50% of the numbers (in binary) starts with "1"
- 66. REAL SLOWNOWREAL SLOWNOW Ina random stream of integers ~50% of the numbers (in binary) starts with "1" 25% starts with "01"
- 67. REAL SLOWNOWREAL SLOWNOW Ina random stream of integers ~50% of the numbers (in binary) starts with "1" 25% starts with "01" 12,5% starts with "001"
- 68. REAL SLOWNOWREAL SLOWNOW Ina random stream of integers ~50% of the numbers (in binary) starts with "1" 25% starts with "01" 12,5% starts with "001" If you observe a random stream and see a "001", there is a higher chance that this stream has a cardinality of 8.
- 69. BUCKETINGBUCKETING number: 13,200,393 hash: 2,005,620,294 bits:[100010110101011001000110] 100010110101011001 000110 ---------------|------ value index
- 70. BUCKETINGBUCKETING number: 13,200,393 hash: 2,005,620,294 bits:[100010110101011001000110] value: number of zeros +1 (rtl) 100010110101011001 000110 ---------------|------ value index
- 71. BUCKETINGBUCKETING number: 13,200,393 hash: 2,005,620,294 bits:[100010110101011001000110] value: number of zeros +1 (rtl) index: The lowest b bits used to determine the index of the register whose value is to be updated. (m=2b) 100010110101011001 000110 ---------------|------ value index
- 72. BUCKETINGBUCKETING number: 13,200,393 hash: 2,005,620,294 bits:[100010110101011001000110] value: number of zeros +1 (rtl) index: The lowest b bits used to determine the index of the register whose value is to be updated. (m=2b) each bucket will serve as an "estimator" 100010110101011001 000110 ---------------|------ value index
- 73. ESTIMATINGESTIMATING LogLog: In order tocompute the number of distinct values in the stream you would just take the average of all of the m buckets distinct vals = constant * m * 2^(avg R)
- 74. ESTIMATINGESTIMATING LogLog: In order tocompute the number of distinct values in the stream you would just take the average of all of the m buckets HyperLogLog uses large range correction (??) Harmonic Mean which tends to behave better for extreme values distinct vals = constant * m * 2^(avg R)
- 75.
- 76.
- 77. UNIONS/INTERSECTIONSUNIONS/INTERSECTIONS Are lossless (forsame HLL size) Some guys tried to combine different HLL with different sizes How many distinct visitors we had in Monday AND Tuesday?
- 78. PARAMETERSPARAMETERS number of buckets/registers theoreticalHLL error bounds (1.04 / sqrt(m))
- 79. THIS IS HUGETHISIS HUGE Who's using? Node, Java, C, etc etc Postgres Redis Twitter Algebird, Scalding Druid (MPP) Basically anyone who needs to count distinct/group-by
- 80.
- 81. CONCLUSIONCONCLUSION evaluate the scenarioto see if approximation is useful
- 82. CONCLUSIONCONCLUSION evaluate the scenarioto see if approximation is useful test the npm packages, some are just crap
- 83. CONCLUSIONCONCLUSION evaluate the scenarioto see if approximation is useful test the npm packages, some are just crap I would write them in C and use f
- 84.