Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c49cb2d
Add tools/vocab/familiar_candidates.py, drawing familiar forms from O…
derek73 Nov 13, 2026
5d7c583
Count familiar_candidates.py's titles read in the layering control
derek73 Nov 13, 2026
cca7ea0
Keep spellings and transliterations in familiar_candidates.py for the…
derek73 Nov 13, 2026
631b70a
Merge master into feat/familiar-forms-cc0
derek73 Nov 13, 2026
7ededad
WIP: the spelling kind in the reading index and the lexicon
derek73 Nov 13, 2026
346833a
Ship the reviewed familiar forms and spellings, the spelling a kind o…
derek73 Nov 13, 2026
cadc298
WIP: the alternate-spelling outcome
derek73 Nov 13, 2026
bd3d0b9
WIP: grid witnesses whose letters differ
derek73 Nov 13, 2026
451e6f5
WIP: alternate-spelling docs and controls
derek73 Nov 13, 2026
c88dc54
WIP: decisions entry for alternate spellings
derek73 Nov 13, 2026
1e3b46b
WIP: the eight translation splits decided
derek73 Nov 13, 2026
53d4321
Fold the letters a keyboard lacks as one types them in the name match
derek73 Nov 13, 2026
e27bb6a
Accept a name needing two leniencies only where the caller accepts both
derek73 Nov 13, 2026
8016967
Read a standard transcription of a letter a keyboard lacks as folded …
derek73 Nov 13, 2026
f1c3f19
WIP: transcription rows
derek73 Nov 13, 2026
10e9c6c
Ship no reading for a spelling the match's fold now reads folded
derek73 Nov 13, 2026
87806b1
WIP: decisions entries for the fold rules and alternate spellings, co…
derek73 Nov 13, 2026
01be1d6
WIP: the match_diff figures
derek73 Nov 13, 2026
492e61d
Read a transcription letter by letter and on every side of a comparison
derek73 Nov 13, 2026
944e5c9
WIP: the fold entry after review
derek73 Nov 13, 2026
8e50c12
WIP: the retaken match_diff figures
derek73 Nov 13, 2026
87b1d5c
Index the pairs under every form, and cut one key per form set
derek73 Nov 13, 2026
db353f3
WIP: the fold entry after the second review
derek73 Nov 13, 2026
e52740a
WIP: the third match_diff figures
derek73 Nov 13, 2026
6d60d6f
Read the pipeline's hyphens alike in the pair index and the match
derek73 Nov 13, 2026
665d850
Merge master into feat/familiar-forms-cc0
derek73 Nov 13, 2026
d9c2c91
Record the collapsed keys as decided, with their measured cost
derek73 Nov 13, 2026
901498a
Cover the forms bound and the familiar outcome's no-pair guard
derek73 Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o

**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/<tag>.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` runs the build checks that replaced the modules' import-time asserts (its docstring numbers them), turns every reading into an index row with its file's tag, and writes the rows, and every file's tag (`FILES`, a file holding only exclusions included, for a strict choice to name), as the committed reading index `nameparser/config/_index.py` (#753, phase 3a) and, compiled, as the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line; the build never reads `_index.py` back. A file a shipped pack reads may also hold a `[rules]` table, each key a `Policy` field (`patronymic_rules`, `name_order`, `two_family_names`, `two_given_names`, `initials_as_family`), which the build checks against its `RULE_KEYS` and `PACK_RULES` and writes as `nameparser/config/_rules.py`; `_locale.rules_patch(tag)` turns a file's table into the `PolicyPatch` a pack carries, RU's and TR_AZ's from `ru.toml` and `tr.toml`, and the `two_family_names` of ES, CS and SK from `es.toml`, `cs.toml` and `sk.toml`, and ES_AR's from `es.toml`, a country pack reading its language's table (#754, 4f; decisions/vocabulary-model's 2026-10-29 4f entry), and VI's `name_order` from `vi.toml` (#1108), and NE's `initials_as_family` from `ne.toml` (#1254). The compiler is `nameparser/_reading_index.py`'s `compile_readings`, with the kind and property tables it reads (`CATEGORY`, `BEHAVIORAL`, `GATED_WORDS`), so the build and a choice of languages compile alike: `Locale.languages` (3.0, phase 3b) is a choice of BCP 47 tags, which `parser_for` unions across its packs and base's recorded choice and records on the `Lexicon` (`Lexicon.languages`, 3.0's 4i-c), the lexicon compiling its base's readings through `choose` and its own edits after them, `choose` selecting the readings (a chosen file's `default = false` ones on; on a word a chosen file holds a vocabulary reading of, other files' vocabulary readings off, save `mul.toml`'s and `und.toml`'s; a chosen conjunction's `may_be_initial` dropped; under `parser_for(..., strict=True)`, 3j, only the chosen files, `mul.toml` (#1005) and native-script words' readings as a plain choice of the same languages keeps them, name readings included, compiled, through `strict_pool`), `changed_words` deriving a choice's deviation surface which tests/v2/test_locales.py's gate reads, and `strict_ungated` the words a strict choice ungates against the plain one, which tests/v2/test_reading_index.py holds to a reviewed table (decisions/vocabulary-model, 2026-10-29 3b; 2026-10-31 #753, phase 3j); the build keeps the checks and loads that module by path, as it loads `_lexicon.py`, and `tests/v2/test_reading_index.py` holds that the index compiled with no choice is `_compiled.py` exactly. `readings_of(text)` there is the run-time reader of the index, every reading of a word with its file's tag, loaded on first call; `languages_of(text, categories)` is the reader a default parse calls for `Reading.languages` (3.0), through the lexicon's private `_files` (`_lexicon._ReadingFiles`, one per base, log and choice, hashed by identity so assemble's memo keyed by it costs no frame), which under a choice or a caller's log names the files whose reading is on in that lexicon instead (3.0's 4i-d), so the first parse that reads a word from its `Lexicon`'s vocabulary loads the index. Through 2.x each `config/` module re-exported its constants from `_compiled.py` (`TITLES = _compiled.TITLES`) under an autodoc docstring; 3.0 removed those modules (#1007, 2.8.1 warning of them), so the compiled sets are private, `Lexicon` reads them, and a caller lists a kind's words with `Lexicon.words(kind=...)`, docs/migrate.rst giving the call for each 2.x constant (a test runs every one). `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, and `tests/v2/test_reading_index.py` when the index is, each naming the refresh command, and `test_vocab.py` gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model.

The compiled SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring a `Lexicon`, never by editing a constant, and the shipped default is changed by editing the TOML and rebuilding. **The four dicts are not frozen.** `_compiled.CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped, private since 3.0, and read once by the cached `Lexicon.default()`, so an edit after the first parse misses it — configure the object (`dataclasses.replace(lexicon, capitalization_exceptions=...)`). `_compiled.SAME_AS` (the name match's listed spellings, a dict of dicts, below), `_compiled.SAME_AS_NAMED` beside it and `_compiled.FAMILIAR` (the familiar forms) are the same hazard: each `Lexicon` reads them once into its own `_same_as`, `_same_as_named` and `_familiar` at construction, the default's cached, so an edit to the module misses every lexicon already built — configure the object (`lexicon.add(word, kind=..., same_as=...)`). `REGEXES`, the other mutable mapping through 2.x, went with `config/regexes.py` in 3.0 (#1008). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` sweeps every module in `nameparser/config/` (`_compiled` and `_ko_surnames` hold its constants) and names those four dicts as explicit exemptions rather than letting its `isinstance` filter drop them.
The compiled SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring a `Lexicon`, never by editing a constant, and the shipped default is changed by editing the TOML and rebuilding. **The five dicts are not frozen.** `_compiled.CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped, private since 3.0, and read once by the cached `Lexicon.default()`, so an edit after the first parse misses it — configure the object (`dataclasses.replace(lexicon, capitalization_exceptions=...)`). `_compiled.SAME_AS` (the name match's listed spellings, a dict of dicts, below), `_compiled.SAME_AS_NAMED` beside it, `_compiled.FAMILIAR` (the familiar forms) and `_compiled.ALTERNATE_SPELLINGS` (the alternate spellings) are the same hazard: each `Lexicon` reads them once into its own `_same_as`, `_same_as_named`, `_familiar` and `_alternate` at construction, the default's cached, so an edit to the module misses every lexicon already built — configure the object (`lexicon.add(word, kind=..., same_as=...)`). `REGEXES`, the other mutable mapping through 2.x, went with `config/regexes.py` in 3.0 (#1008). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` sweeps every module in `nameparser/config/` (`_compiled` and `_ko_surnames` hold its constants) and names those five dicts as explicit exemptions rather than letting its `isinstance` filter drop them.

The compiled sets, read by `Lexicon` (mostly as the `_sets` field of the same name lower-cased; `GLUED_HONORIFICS` is `honorific_tails`, `NON_GIVEN_NAME_PARTICLES` derives `particles_ambiguous`, and `CAPITALIZATION_EXCEPTIONS` is the `capitalization_exceptions` field); a caller lists one with `Lexicon.words(kind=...)`, or a comprehension over it where the set is derived, docs/migrate.rst giving each:

Expand All @@ -261,7 +261,7 @@ The compiled sets, read by `Lexicon` (mostly as the `_sets` field of the same na
- the views' kinds — `GENERATION_WORDS`, `CREDENTIAL_WORDS` (a credential written as a word), `HONORIFIC_WORDS` and `PATRONYMIC_PARTICLES` (3.0's 4b and 4c), private: the kinds the other sets do not record, which classify reads to tag a word for the `generation`, `credentials`, `honorifics` and `patronymic` views (rules/R8, rules/R9)
- case masks — `CAPITALIZATION_EXCEPTIONS` mapping of case MASKS (e.g. `{'phd': 'PhD'}`), compiled from the `cased` property, at most one per word: each value is its key's own letters and digits recased, laid over the word as written, so `ph.d.` repairs to `Ph.D.` and repair never writes the value's own punctuation into the word (#459); `Lexicon` raises `ValueError` on a value that spells anything else; and `CAPITALIZED_PARTICLES` (the `saint` and `rank` kinds, private), the particles case repair writes with a capital rather than in lowercase, keyed on the kind so a word needs no mask of its own (rules/R4, #1069), and `CAPITAL_OPENING_PARTICLES` (the particle readings written `capital_opening`, `ter` and `der`, private), the particles case repair writes with a capital where the word before it in the family name is no particle, nor, where it opens the family name, the word written right before it in any role (`Levon Ter Petrosyan`, but `Jan van der Berg` and `Van der Walt`), keyed on a property because `der`'s kind, `article`, is `de`'s and `la`'s too (rules/R12)
- listed spellings — `SAME_AS`, private, read by `Lexicon` as `_same_as` and nowhere in a parse: every `same_as` keyed by the group of its reading's kind (`_reading_index.SAME_AS_GROUPS`: `given` for a given name or a bound given name, `family` for a surname or a particle, `title`, `suffix`), each word with every full form it carries there (`sr` among titles is both `señor` and `senhor`). The name match (`_match.py`) reads each role in its group through `Lexicon._headwords`, every word reachable through the group's links, so a given name's spelling never matches a surname (`Ma, Jack` / `María, Jack` conflict) and a loop is simply all its words (decisions/name-matching, 2026-11-08); and `SAME_AS_NAMED`, per group, the words of it whose name reading's `same_as` stands beside a name reading of the same kind with none, a name of its own too (`jo`), whose links `_headwords` follows only where `_match._spellings` read the word with its trailing period (the 2026-11-08 #1006 follow-up)
- familiar forms — `FAMILIAR`, private, read by `Lexicon` as `_familiar` (through `Lexicon._familiar_forms`) and nowhere in a parse: each `familiar` reading's `short_for`, the longer given names a short one stands for (`max` -> `maximilian`, `maxwell`), every file's together, which `Lexicon` reads both ways (a pair matches whichever way its source wrote it, never chained) and the name match reads as the opt-in `familiar-form`. `familiar` is a category of its own, never a `given` reading, which would move parses where the word is also a title, particle or suffix; a choice neither prefers nor drops by it (`_reading_index.UNPREFERRED`), and `tools/vocab/familiar.py` writes the shipped ones, English alone, from the nicknames list, less the pairs `tools/vocab/nicknames_excluded.tsv` refuses (decisions/name-matching, 2026-11-08 FAMILIAR FORMS, 2026-11-09 THE NICKNAMES LIST and 2026-11-12 ENGLISH ONLY UNTIL REBUILT)
- familiar forms — `FAMILIAR`, private, read by `Lexicon` as `_familiar` (through `Lexicon._familiar_forms`) and nowhere in a parse: each `familiar` reading's `short_for`, the longer given names a short one stands for (`max` -> `maximilian`, `maxwell`), every file's together, which `Lexicon` reads both ways (a pair matches whichever way its source wrote it, never chained) and the name match reads as the opt-in `familiar-form`. `familiar` is a category of its own, never a `given` reading, which would move parses where the word is also a title, particle or suffix; a choice neither prefers nor drops by it (`_reading_index.UNPREFERRED`), and `tools/vocab/familiar.py` writes the shipped ones, and the shipped spellings, from the nicknames list, less the pairs `tools/vocab/nicknames_excluded.tsv` refuses, and from `tools/vocab/familiar_reviewed.tsv`, the pairs `tools/vocab/familiar_candidates.py` finds in ORCID as two readers reviewed them (decisions/name-matching, 2026-11-08 FAMILIAR FORMS, 2026-11-09 THE NICKNAMES LIST and 2026-11-12 ALTERNATE SPELLINGS ARE OPT-IN); and `ALTERNATE_SPELLINGS` beside it, each `spelling` reading's `spelling_of`, the given names a name is another spelling of by the person's or a romanization's choice (`olha` of `olga`), read by `Lexicon` as `_alternate` (through `Lexicon._alternate_spellings`) both ways and never chained, and by the name match as the opt-in `alternate-spelling`: a kind of its own, never a `given` reading's `same_as`, which would make the word a name written short and move parses (`GIVEN_NAME_ABBREVIATIONS`)

- the group suggester's tables — `config/_order_names.py`, private, written by `tools/vocab/order_names.py` (`--write`; its hand-kept spelling lists sit above the generated blocks) and read by `_suggest` only, never by a parse or a `Lexicon`: `ROMANIZED_SURNAMES` (zh, ko and vi surnames folded by `_suggest.fold`) and `COMMON_GIVEN_NAMES`. They are not `Lexicon` kinds, so a language choice, `strict=True` and a caller's edit do not reach them and no snapshot moves with them (decisions/group-suggestion, 2026-11-10 TABLES)

Expand Down
Loading
Loading