A CRC32 checksum for python - implemented in Nim lang for lower-level language.
No compilation or setup. Just a micro library for the crc32 function
pip install microcrcUsage:
from microcrc import crc32
result = crc32('Hello World')
1243066710
hex(result)
'0x4a17b156'That's it - use it for anything.
-
Is fast fast fast
Using a compiled nim module under-the-hood we gain c-speed fastness. Pre-computing the CRC table ensures only XOR bit manipulation is the main task.
-
Table-Driven Approach:
Precomputing a lookup table speeds up end-usage, so it's more efficent when running.
-
Made in Nim (exported to Python)
Nim's
nimpymodule makes it simple to expose native Nim code to Python. -
Matches online examples:
Using polynomial
0xEDB88320ensures we match JS and other online examples.
The core function crc32 is written in "nim-lang" and compiled into a .pyd, bundled into the package.
The nim module ./src/crc.nim contains a pre-compiled lookup table and the python exposed proc. To gain more speed the proc uses memory address pointers, using the pre-computed lookup-table when computing the xor.
Benchmarks mean nothing. 99% of the time they only detail the 1% of perfect cases. That said - Checkout this benchmark:
Comparing against the fastest python crc library https://github.com/marzooqy/anycrc
Name Outliers OPS (Kops/s) Rounds
-------------------------------------------------------------------
test_anycrc_32 5906;5906 755.0418 (0.99) 79854
test_crc_32 2497;2497 759.6637 (1.0) 92799
-------------------------------------------------------------------
We see that anycrc is 1% slower!
- Best result operations per second is 1% faster than
anycrc(the fastest in python) - especially from cold-start. - On average it's ~5% slower (AFTER WARMUP!)
- But also;
any_crcis 4% slower than its own fastest run
Name (time in ns) OPS (Kops/s) Rounds Iterations
---------------------------------------------------------------------------------------------------
test_anycrc_32 (NOW) 834.4711 (1.0) 98107 1
test_anycrc_32 (windows-cpython-3.8-64bit/0030_51bb0b1) 802.7725 (0.96) 85845 1
test_crc_32 (NOW) 796.8278 (0.95) 78040 1
test_crc_32 (windows-cpython-3.8-64bit/0030_51bb0b1) 730.7685 (0.88) 98107 1
Caveats:
- speed runs are from cold-start, then 4 hot runs.
anycrcis faster after warm-up. - This is an older version of Nim, Windows, and Python - newer will probably be faster
- anycrc is a much better product
Legend:
test_anycrc_32: theanycrcstandard 32 poly model.test_crc_32: my implementation - same poly.
1.0means the target - as the fastest
0.99defines 1% from the target fastest.
Name Outliers OPS (Kops/s) Rounds
-------------------------------------------------------------------
test_anycrc_32 5906;5906 755.0418 (0.99) 79854
test_crc_32 2497;2497 759.6637 (1.0) 92799
-------------------------------------------------------------------
Name Min Max Mean
--------------------------------------------------------------------------------------
test_anycrc_32 873.0000 (1.0) 80,088.0000 (1.0) 1,324.4300 (1.01)
test_crc_32 873.0000 (1.00) 95,522.0000 (1.19) 1,316.3719 (1.0)
--------------------------------------------------------------------------------------
Name StdDev Median IQR
--------------------------------------------------------------------------------------
test_anycrc_32 676.4560 (1.11) 1,165.0000 (1.0) 291.0000 (1.00)
test_crc_32 611.1147 (1.0) 1,165.0000 (1.0) 291.0000 (1.0)
--------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
General benchmarks:
Name (time in ns) IQR Outliers OPS (Kops/s) Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------
test_anycrc_32 (stored) 291.0000 (>1000.0) 6130;6130 864.9554 (1.0) 85845 1
test_anycrc_32 (NOW) 291.0000 (>1000.0) 1410;2646 859.3654 (0.99) 81760 1
test_crc_32 (stored) 291.0000 (>1000.0) 1786;3553 716.0017 (0.83) 83753 1
test_crc_32 (NOW) 0.0000 (1.0) 1507;23624 790.0587 (0.91) 90359 1
-----------------------------------------------------------------------------------------------------------------------------------
The results are a HEX (octet) numbers and uses the standard crc polynomial. Therefore results are compatible with online examples
Python:
# python
from crc import crc32
v = crc32('The quick brown fox jumps over the lazy dog')
1095738169 # int
hex(v)
'0x414fa339'
oct(v)
'0o10123721471'JS:
// js
let msg = 'The quick brown fox jumps over the lazy dog'
// An exact value from py
python_result = 0x414FA339
// https://www.npmjs.com/package/@tsxper/crc32
const result = crc32(msg)
1095738169 // is 0x414FA339 in integer form.
result == python_result
crc32(msg).toString(16)
'414fa339'