Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f3a90ea
Read every script question from one private table (#1256)
derek73 Nov 13, 2026
b62c4a5
Read Georgian as a script with no case for the name-case contrast (#1…
derek73 Nov 13, 2026
72c6766
Write no initials for the Myanmar script (#1234)
derek73 Nov 13, 2026
e8e200b
Read the dotted-credential shape's script veto from the abbreviation …
derek73 Nov 13, 2026
28f73e9
Write no initials for the Thai script, keeping its period abbreviatio…
derek73 Nov 13, 2026
15c3a81
Record the script-property table as a mechanism and the split it foun…
derek73 Nov 13, 2026
6142362
Record Mtavruli beside Mkhedruli written in Mkhedruli as decided (#1222)
derek73 Nov 13, 2026
b212787
Correct the comments the script-property table left stale (#1256)
derek73 Nov 13, 2026
1226317
Read four more name-level case questions from the script table (#1222)
derek73 Nov 13, 2026
840528e
Widen the script table's guards from the test review (#1256)
derek73 Nov 13, 2026
0405ff6
Record the local corpora's moves for the script table (#1256)
derek73 Nov 13, 2026
84d6555
Cover the script table's empty-span refusal (#1256)
derek73 Nov 13, 2026
b9db338
Answer the docs review: Georgian dotted runs stay initials, the recor…
derek73 Nov 13, 2026
3720c4a
Answer the code review: packs and suggester recorded, table frozen, r…
derek73 Nov 13, 2026
6e1ebf1
Merge remote-tracking branch 'origin/master' into feat/issue-1256-scr…
derek73 Nov 13, 2026
bb606fd
Record the review's two Accepted shapes and pin the last test gaps (#…
derek73 Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2
- **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, kanji, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content.
- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: "<release>"}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname`, `Reading.first_surname`, `ParsedName.order`, `Policy.two_given_names`, `PolicyPatch.two_given_names`, `Policy.lone_name_word`, `PolicyPatch.lone_name_word` and `Locale.languages`, 3.0.0; `Policy.initials_as_family`, `PolicyPatch.initials_as_family`, `Policy.capitals_as_family` and `PolicyPatch.capitals_as_family`, 3.1.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_same_as`, `_same_as_named`, `_familiar`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and is refused since 3.1 with a `ValueError` (`_lexicon._refused_2x`) that tells the layout 3.0 loaded, 2.7's and 2.8's, to re-pickle under 3.0, and any older one to rebuild — that layout a frozen record (`_LAYOUT_3_0_LOADED`), never `_VOCAB_FIELDS`, which a new field joins and every 2.8 pickle would then lack, the translation 3.0 loaded it by being deleted (decisions/vocabulary-model, 2026-11-08 #1003); the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` are the refusal test's input. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because a foreign blob would otherwise load a different lexicon in silence (a snapshot base of a kind this release does not know, an edit `add` would not have written, a list entry the fold changes).
- **One sanctioned global**: the cached default `Parser` (`_parser._default_parser`, which `parse()` and `ParsedName.matches` use). Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the default parser's) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review.
- **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins render's hand-copied patterns to the pipeline's, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception.
- **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins render's hand-copied patterns to the pipeline's, `test_script_properties.py`, which checks each row of the script-property table through the readers of each column that take a word, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception.

## Extension Patterns

Expand Down Expand Up @@ -328,7 +328,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2

**Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` (v1's name for what is `_vocab._EAST_SLAVIC_CYR` since 3.0 removed `config/regexes.py`, #1008) shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185)

**Suffix words vs the initial veto** — single-letter roman numeral suffixes (`i`, `v`) are suffix words but also initial-shaped, so `_vocab.is_suffix_strict` (v1's `is_suffix`) vetoes them through `_vocab.is_initial`. `_vocab.is_suffix_lenient` accepts suffix words without the veto and is only safe in unambiguous positions, after a comma (#136, #144). **The veto is Latin-only since #320**: `is_initial` ANDs the bare shape test (`is_initial_shaped`) with "not in `_policy._NO_INITIALS`", the four scripts that have no initials (Han, Hangul, Hiragana, Katakana), so the single-character CJK suffix words (씨, 様, 氏, 군, 양, 님, 殿) written with a period read as suffixes: `"김민준 씨."` gives suffix `씨.`. `V.`/`I.` still lose to the veto and still need the lenient test.
**Suffix words vs the initial veto** — single-letter roman numeral suffixes (`i`, `v`) are suffix words but also initial-shaped, so `_vocab.is_suffix_strict` (v1's `is_suffix`) vetoes them through `_vocab.is_initial`. `_vocab.is_suffix_lenient` accepts suffix words without the veto and is only safe in unambiguous positions, after a comma (#136, #144). **The veto is Latin-only since #320**: `is_initial` ANDs the bare shape test (`is_initial_shaped`) with "not in a script that writes no initials", the rows of `_policy._SCRIPT_PROPERTIES` whose `writes_initials` is false (Han, Hangul, Hiragana, Katakana, and since #1256 the Myanmar and Thai scripts; decisions/script-properties), so the single-character CJK suffix words (씨, 様, 氏, 군, 양, 님, 殿) written with a period read as suffixes: `"김민준 씨."` gives suffix `씨.`. `V.`/`I.` still lose to the veto and still need the lenient test.

**Comparing against v1 needs `PYTHONSAFEPATH=1` AND a directory outside the worktree** — `uv run --isolated --no-project --with 'nameparser==1.4.0'` still puts the checkout's `nameparser/` ahead of the pinned wheel on `sys.path`, so the "v1" side silently imports the branch and every comparison reports parity. Run it as `cd <scratch dir> && PYTHONSAFEPATH=1 uv run --isolated --no-project --with 'nameparser==1.4.0' python -c "..."`. This produces false confidence rather than an error, so it invalidates results without ever looking wrong. **A `tests/v2/cases.py` row's `parity` means 1.4.0, and nothing checks it** -- the table runs every row against the tree, so a row labelled `parity` that only matches master passes for good. #614 added eight such rows, compared against master; #620 found them by running the wheel this way and relabelled each with the change its reading came from, found by `git bisect run` in a scratch worktree (`git worktree add --detach`, never this one). Run the wheel before writing `parity`.

Expand Down
2 changes: 1 addition & 1 deletion docs/design/decisions/H2-leading-abbreviation-title.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
- An abbreviation of two or more letters ending in its one period, opening the part that carries the given name (the whole name, or the part after a family comma), is a title even unlisted and even where it is suffix vocabulary, and a single-letter initial is not: `Insp. Jane Morse` → title 'Insp.', `Morse, Det. Insp. Jane` → title 'Det. Insp.', `Esq. Smith` → title 'Esq.', `J. Smith` → given 'J.' — 2026-06-30 #288.
- The shape is an unbroken run of letters, so an unlisted abugida abbreviation written with a vowel sign stays a name word and only vocabulary recognizes it — 2026-06-30 #288 (the run was the whole cause until 2026-11-08 #1191, below, found the word with no sign).
- The shape declines a period-marked word carrying a letter of the ten Brahmic scripts from Devanagari to Sinhala (U+0900 to U+0DFF), an Except of rules/H2, so a Latin initial spelled there with no vowel sign is no title: `एम. एस. वर्मा` → given 'एम.', middle 'एस.' — 2026-11-08 #1191.
- The shape declines a period-marked word carrying a script with no initials: `田中. 太郎` → family '田中.', where `Smith. John` keeps title 'Smith.' — 2026-09-10.
- The shape declines a period-marked word carrying a script that writes no period abbreviations (Han, kana, hangul; Thai and Burmese write no initials but are not among them, decisions/script-properties): `田中. 太郎` → family '田中.', where `Smith. John` keeps title 'Smith.' — 2026-09-10.
- C13 wins for a credential opening the part after a family comma (`Smith, Esq. John` → suffix 'Esq. John'), and the shape still reads a title there for a word that starts no S8 run (`Smith, Insp. John`, `Smith, Do. John` → title) — 2026-10-20 #761; its `Smith, Ma. John` example moved to the given name on 2026-10-26 #789.
- A given name written short outranks the shape: a word the vocabulary lists with a `given` reading whose `same_as` is the full name, period-marked where the shape would fire, is the given name and the name reads as it would with the full name there (`Chas. Smith` → given 'Chas.', `Mrs Ma. Liza Cruz` → title 'Mrs', given 'Ma.'). The shipped set is the traditional English abbreviations, the Filipino `ma` and `jo`, and since #1115 Muhammad, Achmad and Francisco written short (`mohd`, `mhd`, `muh`, `moh`, `moch`, `ach`, `fco`; decisions/H6's 2026-11-06 entry); a listed title still wins, and the reading acts nowhere else — 2026-10-26 #789.
- A transliteration digraph declines the shape, an Except of rules/H2: the Latin spellings of one Cyrillic letter that are no title in any corpus read, `Yu`, `Ya`, `Ye`, `Yo`, `Zh`, `Kh`, `Shch` and `Dzh`, in any case, so `Yu. A. Chibrikova` → given 'Yu.', middle 'A.', as `Ю. А. Чибрикова` reads; `Ts`, `Ch` and `Sh`, each also a title written short, keep the shape, and a digraph of another alphabet or a Latin given name written short (`Th.`) is outside the set — 2026-11-09 #1136.
Expand Down
Loading
Loading