Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ jobs:
# Two passes: the suite in parallel (pyproject's addopts), which
# deselects the clock-timed `serial` tests, then those alone, since
# other workers' load fails their timing ratios (tests/conftest.py).
# The frame, call and shift counts in the same module run in the first pass.
if [ "$COVERAGE_ON" = "true" ]; then
uv run pytest --cov=nameparser --cov-report=
uv run pytest -n0 -m serial --cov=nameparser --cov-append --cov-report=xml
Expand Down
8 changes: 5 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ Three committed contributor docs carry the parser's normative rules and their re

# Run all tests (includes --doctest-modules, so doctests in nameparser/ are also run).
# Parallel by default (pytest-xdist, `-n auto --dist worksteal` in pyproject's addopts),
# which DESELECTS the clock-timed `serial` tests (tests/v2/test_benchmark.py) -- run
# them in the second command. `-n0` runs everything serially in one process.
# which DESELECTS the clock-timed `serial` tests (the scaling ratios and the
# thousand-names backstop in tests/v2/test_benchmark.py) -- run them in the second
# command. The module's frame, call and shift counts run in the first.
# `-n0` runs everything serially in one process.
uv run pytest
uv run pytest -n0 -m serial

Expand Down Expand Up @@ -336,7 +338,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2

**`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version".

**Perf regressions are caught by the scaling tests, not the absolute-time ones** — the thousand-names test (`test_a_thousand_names_still_parse_in_reasonable_time`) uses constant-size, delimiter-free input and is structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why).
**Perf regressions are caught by the scaling tests, not the absolute-time ones** — the thousand-names test (`test_a_thousand_names_still_parse_in_reasonable_time`) uses constant-size, delimiter-free input and is structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. The timed rows read the thread's CPU time (`_CLOCK`), not the wall clock, so they hold on a laptop other sessions are loading: measured 2026-11-11 beside 24 busy-loop processes on 12 cores, the wall clock read 27 of 33 ratio rows at or over their 6.0 bound and the CPU clock none. CPU time is not load-immune -- a thread moved to an efficiency core or onto a cold cache is charged the slower run, and a clean ~4.0x row read up to 5.6x (+40%) under that load -- and a ratio can read LOW too, a slow stretch landing on the small input -- so `_ratio` interleaves the two inputs and, where its first seven rounds read within 20% of the bound or over it, takes all 35 and judges the last minimums. Those rounds are not a retry: each only lowers an input's minimum toward its true cost, so the ratio converges on the true one, though not monotonically, which is why it is judged once, at the end, never at the first reading that passes. Never "fix" a flaky timed row by raising its bound or re-running the test until it passes, which lets a real quadratic through. Count instead of timing wherever the question allows (frames, `_ShiftCountingList`), since a count needs no serial pass and no extra rounds. And an ABSOLUTE frame count depends on what the process parsed before it (decisions/parse-cost, 2026-11-09 #1170). The counted guards that run in the parallel suite are ratios, a ±2% band averaged over 50 parses or wide enough for that drift, and exact counts of one function's calls (`only=`/`caller=`, `== 0` up to `== 6`); each passed alone in a fresh process, after the 21,065 tests of `test_cases.py` and `test_match.py` in one, and in three full parallel runs (2026-11-11, #1352). A new exact pin needs the same check, and a new timed test takes the `serial` mark. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why).

**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`), and the extracts of `local_orcid` and `local_patentsview`, streamed through `gzip.open(..., newline="\n")` rather than read whole for their 25M and 5.3M lines, which cuts at `"\n"` only as `jsonl_lines` does; none of the 16 committed JSONL files held one on 2026-10-21.

Expand Down
Loading
Loading