Repository navigation
Speed up number formatting by caching locale and pattern parsing - #1437
Open
DicksonWu654 wants to merge 4 commits into
Open
DicksonWu654 wants to merge 4 commits into
DicksonWu654 wants to merge 4 commits into
Conversation
Every formatting call that receives a locale identifier string parses it again. Cache the parsed tuple in a 1024-entry LRU keyed by the exact (identifier, sep) strings; the result is an immutable tuple, Locale objects are still created per call, and str subclasses and invalid inputs take the uncached path so errors are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Locale.parse() created two helper closures on every call even though they are only needed when the requested locale does not exist. Try the exact locale first; the fallback path is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Custom format strings (e.g. format_decimal(x, "#,##0.00")) are re-parsed on every call. Cache the parsed constructor arguments (all immutable) in a 1024-entry LRU keyed by the pattern string, as babel.dates already does for date patterns. A new NumberPattern is still returned on every call, since NumberPattern is mutable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The quote-removal regex cannot match a string without an apostrophe, which is the common case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A typical
format_decimal(x, locale='de_DE')orformat_currency(x, 'EUR', format='¤#,##0.00', locale=...)call parses the same locale identifier and often the same number pattern every time. This PR removes that repeated work in four small commits. Each one can be reviewed (or dropped) on its own:parse_locale()results. A 1024-entrylru_cachekeyed by the exact(identifier, sep)strings. The result is a tuple of strings.Localeobjects are still created on everyLocale.parse()call, so nothing mutable is shared.strsubclasses and non-string inputs take the uncached path, so their behaviour and error messages don't change. Exceptions are never cached.Locale.parsefallbacks.Locale.parse()created two closures on every call, but they're only needed when the locale doesn't exist. It now triescls(*parts)first. The likely-subtag fallback code is unchanged.babel.dates.parse_pattern, which already keeps a 1024-entrylru_cache. Here the cache holds the parsed constructor arguments, which are all strings, tuples, ints andNone.NumberPatternis mutable, so a new instance is still built on every call. This helps any call that passes a customformat=string. Locale default patterns are already stored asNumberPatternobjects and skip the cache.re.subthat strips pattern quotes can't match without an apostrophe. Skipping it saves a regex call and a lambda on almost every formatted number.Both caches are bounded, and their keys are the function's only inputs. Neither result depends on locale data or global state.
Benchmarks
The workload, for each of 12 locales (en_US, de_DE, fr_FR, ja_JP, ar_EG, hi_IN, pt_BR, ru_RU, es_ES, zh_Hans_CN, fi_FI, de_CH), is:
format_decimalwith the default pattern and with'#,##0.00',format_currencywith the default pattern and with'¤#,##0.00',format_percent,format_compact_decimal, andparse_decimal. That's 84 calls per run. Each sample was a separate process pinned to one core on Linux. I ran master and this branch in alternating order and give the median per-pair ratio:LocaleobjectLocaleobjectEight pairs on 3.12 and 3.15, six on 3.9. Per commit, on the string-locale workload (Python 3.12, 8 pairs each):
parse_localecache 1.056x (1.023–1.090x),Locale.parsereorder 1.030x (1.009–1.036x), number-pattern cache 1.080x (1.063–1.139x), quote check 1.055x (1.020–1.084x). When callers passLocaleobjects,Locale.parsereturns early and only the twonumberscommits apply.Checks
parse_localeandLocale.parse(with and without likely-subtag resolution, both separators) for every sixth CLDR locale plus aliases, POSIX/C, modifiers, encodings and invalid identifiers. It also coveredformat_decimal,format_currency,format_currency(format_type='name'),format_percent,format_scientific,format_compact_decimal,format_compact_currencyandparse_decimalacross those locales, 19 numbers (including NaN, ±Infinity, -0.0 and 40-digit Decimals) and 24 patterns (quotes,'', currency signs, per-mille, exponents, negative subpatterns and invalid patterns). Each case ran twice so that cache hits were compared too, and I mutated a returnedNumberPatternbetween parses. That's 452,106 results, 8,776 of them exceptions, all identical on Python 3.9, 3.12 and 3.15.LocaleorNumberPatterndoesn't affect the next call, and pickling still works. They also coverstrsubclasses, unhashable and invalid inputs (same exceptions as before), the cache size limits,Localesubclasses whose constructors return other objects or raise, and quote handling after pattern mutation and currency-symbol replacement.pre-commit run --all-filespasses.Overlap: #1380 rewrites the quote cleanup that commit 4 touches. If #1380 is merged first, commit 4 can be dropped or redone on top of it. #1382 and #1370 change
parse_patterninternals; they apply cleanly on this branch and work with the cache.🤖 Generated with Claude Code