Repository navigation
Allow applications to tune the condition that triggers a GIL release and implementation choice in hashlib #91331
Description
| BPO | 47175 |
|---|---|
| Nosy | @gpshead |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-03-30.21:28:38.739>
labels = ['type-feature']
title = 'Allow applications to tune the condition that triggers a GIL release and implementation choice in hashlib'
updated_at = <Date 2022-03-30.21:28:38.739>
user = 'https://github.com/gpshead'bugs.python.org fields:
activity = <Date 2022-03-30.21:28:38.739>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2022-03-30.21:28:38.739>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47175
keywords = []
message_count = 1.0
messages = ['416401']
nosy_count = 1.0
nosy_names = ['gregory.p.smith']
pr_nums = []
priority = 'low'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47175'
versions = []
Activity
gpshead commentedon Mar 30, 2022
Background
All
hashlibcomputations andbinascii.crc32andzlib.crc32release the GIL around their computational core. But they use a hard coded length check to determine when to do so, or always do it.That already accomplishes the larger good of releasing the GIL on big computations. But _probably_ just serves to slow down smaller ones when a GIL release adds more overhead than a context switch to another thread could meaningfully provide in terms of forward progress.
Desire 1
Determine if a threshold should exist at all (should we just always release the GIL for these?) and if so, allow it to be tuned on a per algorithm basis.
This comes at the same time as other enhancements like bpo-47102 and its windows and macos cousins could shift us towards using OS kernel APIs for a subset of algorithms where available - which may effectively "always release" the GIL on OS APIs that are virtual IO call based such as bpo-47102's.
Desire 2
When multiple implementations of an algorithm may be available, allow the user to configure data length thresholds for when each one is triggered. Without meaningfully slowing most things down by adding such logic. Different implementations have different setup and finalization time which can make them more useful for large data rather than tiny.
I'm marking this low priority as it veers towards over-optimization. :) Creating benchmarks and a thing to live in Tools/ that people could run on their target platform to provide a tuning suggestion of what thresholds work best for their needs.
Related inspiring work: OSes often benchmark several algorithm implementations up front to pick a "best" to use for a given platform (ex: see what the Linux kernel does for hashes and raid algorithms). This extends that idea and acknowledges latency as important, not exclusively thru-put.