Repository navigation
Conversation
- browser_profiler: the tasklist probe decoded ANSI/OEM output with the strict default codec; under UTF-8 mode (the default from Python 3.15, PEP 686) the probe raises on CJK Windows and escapes its except tuple, making a live builtin browser look dead. Decode tolerantly - the probe only searches for the ASCII PID digits, which survive lossy decoding. - browser_manager: decode Chromium stdout/stderr launch diagnostics tolerantly so localized early-exit text cannot crash the startup check. - async_url_seeder: pin UTF-8 for the Common Crawl URL cache so non-ASCII URLs round-trip on any system locale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three spots used the platform default codec for text IO, which breaks under non-UTF-8 locales (cp1252/cp936) and under UTF-8 mode — the default from Python 3.15 (PEP 686). Fixes #2439.
BrowserProfiler._is_browser_running(tasklist probe,text=Truewithout encoding)UnicodeDecodeErroron ANSI/OEM output escapes theexcept (ProcessLookupError, PermissionError, OSError)tuple → a live builtin browser is treated as dead, kill/skip logic degrades, processes leakbytes.decode())Failed to start browser_from_cc,aiofiles.openwithout encoding)UnicodeEncodeErrormid-generator under cp1252/cp936; cache files mojibake across localesChanges
browser_profiler.py: the tasklist probe getsencoding="utf-8", errors="replace"— it only searches for the ASCII PID digits, which survive lossy decoding.browser_manager.py: the five diagnostic decodes geterrors="replace"(valid-UTF-8 output is unchanged).async_url_seeder.py: both Common Crawl cache opens pinencoding="utf-8".Testing
New
tests/unit/test_non_utf8_locale_hardening.py(5 tests):aiofiles.openwrapper injects the locale codec only where the caller did not pin one, so the old code fails withUnicodeEncodeErroron any platformVerified locally on a cp936 Windows machine with UTF-8 mode on: 5/5 pass with this patch; 4/5 fail against unpatched develop — including a live
UnicodeDecodeErrorfrom the realtasklistprobe and the startup-check decode crash.Related open PRs in the same theme, with disjoint files/hunks: #1807 (CLI file operations), #2406 (seeder head metadata), #2418 (bundled JS/c4a scripts).