Download to read offline
![Howtoimplementsets?
hashtables( map[int]bool )
bitmap:willf/bitset
compressedbitmaps
Fastindexeswithroaring-DanielLemire#gomtl-10November19th.](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-5-2048.jpg)
![Hashtables
bitmap := map[int]bool{}
for v := 0; v <= 1000000; v += 100 {
bitmap[v] = true
}
Memoryusageperentry?
Fastindexeswithroaring-DanielLemire#gomtl-10November19th.](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-6-2048.jpg)
![in-orderaccessiskindofterrible
[15,3,0,6,11,4,5,9,12,13,8,2,1 ,14,10,7]
[15,3,0,6,11,4,5,9,12,13,8,2 ,1,14,10,7]
[15,3 ,0,6,11,4,5,9,12,13,8,2,1,14,10,7]
[15,3,0,6,11,4 ,5,9,12,13,8,2,1,14,10,7]
[15,3,0,6,11,4,5 ,9,12,13,8,2,1,14,10,7]
[15,3,0,6 ,11,4,5,9,12,13,8,2,1,14,10,7]
Fastindexeswithroaring-DanielLemire#gomtl-10November19th.](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-8-2048.jpg)
![Manipulateabitmap
64-bitprocessor.
Given x ,wordindexis x/64 andbitindex x % 64 .
add(x) {
array[x / 64] |= (1 << (x % 64))
}
Fastindexeswithroaring-DanielLemire#gomtl-10November19th.](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-10-2048.jpg)
![Howfastisit?
index = x / 64 -> a shift
mask = 1 << ( x % 64) -> a shift
array[ index ] |- mask -> a OR with memory
Onebitevery cyclesbecauseofsuperscalarity≈ 1.65
Fastindexeswithroaring-DanielLemire#gomtl-10November19th.](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-11-2048.jpg)
![for v := 0; v <= 1000000; v += 100 {
add v to set
}
bytes/value
map[int]bool 18bytes
willf.BitSet 22bytes
roaring 2bytes
Roaringbitmaps](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-17-2048.jpg)
![Hybridmodel
Setofcontainers
sortedarrays({1,20,144})
bitset(0b10000101011)
runs([0,10],[15,20])
Roaringbitmaps](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-20-2048.jpg)
![Goisshyaboutinlining
Won'tinlinesomesmallfunctionsthatcontainabranch?
func (b *BitSet) Set(i uint) *BitSet {
b.extendSetMaybe(i)
b.set[i>>log2WordSize] |= 1 << (i & (wordSize - 1))
return b
}
https://lemire.me/blog/2017/09/05/go-does-not-inline-functions-when-it-should/
Roaringbitmaps](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-26-2048.jpg)
![Castingasliceistricky
func byteSliceAsUint16Slice(slice []byte) (result []uint16) { // here we create a new slice holder
if len(slice)%2 != 0 {
panic("Slice size should be divisible by 2")
}
// reference: https://go101.org/article/unsafe.html
// make a new slice header
bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice))
rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result))
// transfer the data from the given slice to a new variable (our result)
rHeader.Data = bHeader.Data
rHeader.Len = bHeader.Len / 2
rHeader.Cap = bHeader.Cap / 2
// instantiate result and use KeepAlive so data isn't unmapped.
runtime.KeepAlive(&slice) // it is still crucial, GC can free it)
// return result
return
}
Roaringbitmaps](https://image.slidesharecdn.com/go10roaring-191115023900/75/Fast-indexes-with-roaring-gomtl-10-29-2048.jpg)
Uploaded byDaniel Lemire
Fast indexes with roaring #gomtl-10
Le document présente les index rapides en utilisant les bitmaps Roaring, en discutant des performances optimes pour la manipulation des ensembles d'entiers. Il met en évidence les avantages des bitmaps compressés pour la mémoire et les opérations telles que l'intersection et l'union. Les Roaring bitmaps sont souvent employés dans des systèmes comme Apache Lucene et Elasticsearch pour gérer efficacement de grandes quantités de données.
Fast indexes with roaring #gomtl-10
- 1.
- 2.
- 3.
- 4. tests: ? intersections: ,unions:,differences: Similarity(Jaccard/Tanimoto): Iteration x ∈ S S ∩2 S1 S ∪2 S1 S ∖2 S1 ∣S ∩1 S ∣/∣S ∪1 1 S ∣2 Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 5.
- 6. Hashtables bitmap := map[int]bool{} forv := 0; v <= 1000000; v += 100 { bitmap[v] = true } Memoryusageperentry? Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 7.
- 8. in-orderaccessiskindofterrible [15,3,0,6,11,4,5,9,12,13,8,2,1 ,14,10,7] [15,3,0,6,11,4,5,9,12,13,8,2 ,1,14,10,7] [15,3,0,6,11,4,5,9,12,13,8,2,1,14,10,7] [15,3,0,6,11,4 ,5,9,12,13,8,2,1,14,10,7] [15,3,0,6,11,4,5 ,9,12,13,8,2,1,14,10,7] [15,3,0,6 ,11,4,5,9,12,13,8,2,1,14,10,7] Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 9. Bitmaps Efficientwaytorepresentsetsofintegers. Forexample,0,1,3,4becomes 0b11011 or"27". 0b00001 0b01001 0b11001 0b11011 {0}→ {0, 3} → {0, 3, 4} → {0, 1, 3, 4} → Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 10. Manipulateabitmap 64-bitprocessor. Given x ,wordindexisx/64 andbitindex x % 64 . add(x) { array[x / 64] |= (1 << (x % 64)) } Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 11. Howfastisit? index = x/ 64 -> a shift mask = 1 << ( x % 64) -> a shift array[ index ] |- mask -> a OR with memory Onebitevery cyclesbecauseofsuperscalarity≈ 1.65 Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 12. Bitparallelism Intersectionbetween{0,1,3}and{1,3} asingleANDoperation between 0b1011 and0b1010 . Resultis 0b1010 or{1,3}. Nobranching! Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 13.
- 14. Git(GitHub)useEWAH Run-lengthencoding Example: est Codelongrunsof0sor1sefficiently. https://github.com/git/git/blob/master/ewah/bitmap.c 000000001111111100 00000000 −11111111 − 00 Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 15. Complexity Intersection: or In-placeunion( ):or O(∣S ∣ +1 ∣S ∣)2 O(min(∣S ∣, ∣S ∣))1 2 S ←2 S ∪1 S2 O(∣S ∣ +1 ∣S ∣)2 O(∣S ∣)2 Fastindexeswithroaring-DanielLemire#gomtl-10November19th.
- 16.
- 17. for v :=0; v <= 1000000; v += 100 { add v to set } bytes/value map[int]bool 18bytes willf.BitSet 22bytes roaring 2bytes Roaringbitmaps
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. Goisshyaboutinlining Won'tinlinesomesmallfunctionsthatcontainabranch? func (b *BitSet)Set(i uint) *BitSet { b.extendSetMaybe(i) b.set[i>>log2WordSize] |= 1 << (i & (wordSize - 1)) return b } https://lemire.me/blog/2017/09/05/go-does-not-inline-functions-when-it-should/ Roaringbitmaps
- 27.
- 28. 0x1093534 0fb63d22810c00 MOVZX0xc8122(IP), DI 0x109353b 4084ff TESTL DI, DI 0x109353e 7407 JE 0x1093547 0x1093540 f3480fb8f6 POPCNT SI, SI 0x1093545 ebd6 JMP 0x109351d 0x1093547 4889442418 MOVQ AX, 0x18(SP) 0x109354c 4889542410 MOVQ DX, 0x10(SP) 0x1093551 48894c2420 MOVQ CX, 0x20(SP) 0x1093556 48893424 MOVQ SI, 0(SP) 0x109355a e801ffffff CALL math/bits.OnesCount64(SB) 0x109355f 488b742408 MOVQ 0x8(SP), SI 0x1093564 488b442418 MOVQ 0x18(SP), AX 0x1093569 488b4c2420 MOVQ 0x20(SP), CX 0x109356e 488b542410 MOVQ 0x10(SP), DX 0x1093573 488b5c2440 MOVQ 0x40(SP), BX 0x1093578 eba3 JMP 0x109351d Roaringbitmaps
- 29. Castingasliceistricky func byteSliceAsUint16Slice(slice []byte)(result []uint16) { // here we create a new slice holder if len(slice)%2 != 0 { panic("Slice size should be divisible by 2") } // reference: https://go101.org/article/unsafe.html // make a new slice header bHeader := (*reflect.SliceHeader)(unsafe.Pointer(&slice)) rHeader := (*reflect.SliceHeader)(unsafe.Pointer(&result)) // transfer the data from the given slice to a new variable (our result) rHeader.Data = bHeader.Data rHeader.Len = bHeader.Len / 2 rHeader.Cap = bHeader.Cap / 2 // instantiate result and use KeepAlive so data isn't unmapped. runtime.KeepAlive(&slice) // it is still crucial, GC can free it) // return result return } Roaringbitmaps
- 30.