Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ More shapes the parser reads:
>>> ivan.given, ivan.patronymic, ivan.family
('Иван', 'Петрович', 'Сидоров')
>>> locales.available()
('bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'hi', 'hr', 'ja', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh')
('bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'hi', 'hr', 'ja', 'mn', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh')

When a reading is wrong for your data
-------------------------------------
Expand Down
5 changes: 4 additions & 1 deletion docs/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,10 @@ by ``tests/v2/test_locales.py``:
``sl``, ``hr``, ``ga``, ``gd``, ``es``, ``es_ar``, ``pt``, ``pt_br``, ``vi``, and ``ja``'s 君): a name for each word the choice changes, in each script it
is written in and each position where the default reads it
otherwise — ``test_choice_rotators_cover_every_changed_word`` fails
until every word is held.
until every word is held. A pack whose choice changes no word
beside a patch that does (``mn``, whose file's readings are all on
by default) has no word to hold: list it in the test file's
``_INERT_CHOICES``, and its rotators cover what the patch moves.
- **With no vocabulary at all**: list the shapes its *segmenter* must
divide instead, and mark the rotator tests to skip when the
optional dependency is absent, so the contract tests still run
Expand Down
12 changes: 11 additions & 1 deletion docs/design/decisions/W4-script-scoped-order.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@

Elsewhere: the katakana middle dot divides like a space and its pieces read by the script license (`威廉・莎士比亚` → family '威廉'), decisions/T3's 2026-07-30 #298 entry on codepoint scope; a lone word in a script with no order of its own still reports `given-or-family` (`マイケル`), decisions/O5's 2026-09-07 #449 entry.

Open: none in this file.
- A pack may carry a script order for a script that settles no order, licensed by the caller's choice of the pack and never a default: `locales.MN`'s Cyrillic entry reads `Батын Сүхээ` family 'Батын' and leaves `Batyn Sükhee` and a lone `Сүхээ` to the declared order and O5; `PolicyPatch.script_orders` composes by script — 2026-11-12 #1237.

Open: whether an O1 ending or an O2 marker the active rule reads should keep its name from a pack's script order (the 2026-11-12 #1237 entry's Accepted bullet); no issue filed.

## Log

Expand All @@ -21,3 +23,11 @@ Open: none in this file.
- 2026-10-03 #594 (its voicing-mark argument against an NFKC fold superseded by #596 below) — HALFWIDTH KATAKANA IS KATAKANA. The script table classifies the halfwidth kana block U+FF65–FF9F as `Script.KATAKANA`, so `山田 タロウ` reads family 山田 as `山田 エミ` does; every release through 2.3.0 left the block unclassified and read it given 山田, family タロウ, the positional reading. Legacy data written for JIS X 0201 systems, which had no kanji, spells names this way — bank, payroll and CSV exports. DECIDED by widening the table rather than by folding: nothing rewrites the text before parsing (rules.md's T Background), so normalization at tokenize was never available, and an NFKC fold at classification — the other place the issue offered — reaches far past the kana (`JOHN` to `JOHN`, `㈱` to `(株)`), and it would move a wholly-katakana name across W4's line: a voicing mark with no precomposed form to join (`ア゙`, `ン゙`) folds to the COMBINING U+3099, which sits in the HIRAGANA block, so the folded token reads as kana-licensed Japanese and `ア゙イ タロウ` would read family-first where the unfolded one keeps the declared order (measured 2026-10-03 by parsing the NFKC spelling `ア゙イ タロウ`, which reads family `ア゙イ`; a two-input sweep of halfwidth names against their NFC(NFKC) spellings differed only where such a mark stood; RECOMPUTE: parse each halfwidth name and `unicodedata.normalize("NFC", unicodedata.normalize("NFKC", name))` and compare the fields). The interpunct flank guard is NOT the hazard, though an earlier wording of this bullet said it was: it reads raw text and asks only whether a character is classified at all, which hiragana and katakana both are. The same measurement shows a pre-existing gap the fold would have widened rather than created: full-width katakana typed with an uncomposable combining U+3099 already takes the kana license today. A block range is how the table already decides (U+30FB and the voicing marks are in by block), and it takes the voicing marks U+FF9E/U+FF9F in because they are spacing characters following their base. The span starts at U+FF65, the halfwidth nakaguro, which tokenize already treats as a separator (rules/T2), and stops short of U+FF61–FF64, whose U+FF61 is a full stop. Everything keyed on the table follows: the kana license, the second-East-Asian-word count that keeps a segmenter from being asked (`高橋一郎 タロウ`), the 间隔号's flank guard (`タロウ·ヤマダ` divides), and `_NO_INITIALS`, which closes the limit #cjk-full-stops recorded (`タナカ. John` read title `タナカ.`). THE RATIONALE FOR WHOLLY-KATAKANA NAMES IS AMENDED, NOT THE BEHAVIOR: a wholly-halfwidth name keeps the declared order exactly as a wholly-katakana one does (`ヤマダ タロウ` reads given ヤマダ), but the 2026-07-27 reason — that such names are predominantly transcriptions — is weaker here, since a system with no kanji wrote Japanese names in katakana too, family-first. What stands is the narrower claim: the script cannot tell a Japanese reading from a transcription, so it settles no order, and a caller who knows their katakana is Japanese adds `(Script.KATAKANA, FAMILY_FIRST)` to `script_orders` (docs/usage.rst's Boundaries section carries the recipe). `name_order=FAMILY_FIRST` works too, at the cost of reversing Latin names as well. MEASURED 2026-10-03: the differential gate saw nothing before this change because no corpus line held halfwidth kana; the eight case rows #594 added are its whole population, five of them contract movers (`山田 タロウ`, `山田 ダイスケ`, `山田 ペーター`, the unspaced `山田タロウ`, `タロウ·ヤマダ`), two tolerated radar movers (`タナカ. John`, `マイケル.`) and one parity row (`ヤマダ タロウ`). RECOMPUTE: parse the eight on this tree and on 2.3.0 from PyPI and read `title` beside the name fields.
- 2026-10-03 #596 — A VOICING MARK TAKES THE SCRIPT OF THE KANA BEFORE IT. The four kana voicing marks U+3099–309C are in the hiragana BLOCK, which is how the script table decides, but none belongs to either syllabary: a mark voices the kana it follows. NFC composes one only where a precomposed voiced kana exists (カ + U+3099 is ガ), so `ア゙イ`, `ン゙` and every spacing ゛/゜ kept a hiragana-block character inside a katakana word, the word took the kana license, and a wholly-katakana name read family-first — the gap the #594 entry above measured, present since 2.1.0. DECIDED by deleting, from the classification copy only, every mark standing after a KANA, so the word reads as its base's script; token text is untouched. A mark after anything else — hangul, Han, Latin, or nothing — voices no kana and keeps the table's answer, as before. That scope is a review finding, not a nicety: the first cut deleted every mark with ANY character before it, which handed `김゙민준` the hangul script, and the surname division, which cuts the raw text, then split the mark off its base — given 김, family `゙민준`, unreported, where 2.3.0 left the word whole and reported given-or-family; `parser_for(locales.ZH)` cut `王゙小明` the same way. A case row watches `김゙민준` — marked tolerated, the reading of a mark after a non-kana being unpromised (rules.md's W Background) — so the radar tier, not the contract, sees it move. Declined: taking the marks out of the hiragana range instead. Measured 2026-10-03 on patched copies of this tree, it fixes the katakana names as well but leaves a word holding a mark in NO script, so `あ゙い たろう` loses family-first, `山田 ア゙イ` loses the kana license and reads family `ア゙イ`, and the unspaced `山田ア゙イ` reads given; the two parity case rows #596 added pin those readings. Declined too: narrowing to the two combining marks, which leaves `ア゛イ タロウ` family-first. NFD-typed text was the issue's worry and is not affected by either option, since classification composes first: NFD `やまだ がくと`, `ジョン ガイ` and `김 민준` parse identically under all of them. A mark with nothing before it keeps the table's answer, so `゙アイ タロウ` still reads family-first; that is degenerate input, recorded in rules.md's W Background as unpromised rather than given a branch. ONE ARGUMENT IN THE #594 ENTRY ABOVE NO LONGER HOLDS: an NFKC fold would no longer move a wholly-katakana name across W4's line, since the combining mark it produces (`ア゙` to `ア゙`) now reads as the katakana it follows — #594's own RECOMPUTE, run 2026-10-03 on this tree over `ア゙イ タロウ`, `ン゙ タロウ`, `山田 ア゙イ`, `マイケル ア゙イ`, `ア゚イ タロウ`, `山田 タロウ` and `ヤマダ タロウ` under all three orders, finds no role differing. The range decision stands on its other ground, the fold's reach past the kana (`JOHN`, `㈱`). The interpunct clause of rules/W4 was corrected in the same change: a name the 间隔号 divides keeps the DECLARED order, not its source order — under `name_order=FAMILY_FIRST` `威廉·莎士比亚` reads family 威廉 — and an example pins it. MEASURED 2026-10-03: no corpus line held an uncomposable mark before this change, so the case rows #596 added are the whole population. At 2.1.0, 2.2.0 and 2.3.0 the three `fix(#596)` rows move; at 2.0.0, 2.1.0 and 2.2.0 `김゙민준` moves in its report alone, #449's given-or-family being new in 2.3, and is parity at 1.4.0, which has no report. At 1.4.0 and 2.0.0, which had no kana license, the three movers are parity, while `あ゙い たろう` and `山田 ア゙イ` move as every native-script CJK name does there, under the existing `fix(#271/#272/#298)` rule. RECOMPUTE: parse `ア゙イ タロウ`, `ア゛イ タロウ`, `マイケル ア゙イ`, `あ゙い たろう`, `山田 ア゙イ` and `김゙민준` here and on each baseline from PyPI.
- 2026-10-20 (Derek), #761 — the interpunct stand-down moves from Accepted to Except in the skeleton migration (PR #782): a name the interpunct divides is inside When and Then stops holding there, which is the format's definition of an exception. Words and examples unchanged.
- 2026-11-12 #1237 — A PACK MAY CARRY A SCRIPT ORDER FOR A SCRIPT THAT SETTLES NO ORDER, LICENSED BY THE CALLER'S CHOICE OF THE PACK, NEVER A DEFAULT: `Script.CYRILLIC` JOINS THE ENUM AND THE TABLE, AND ONLY `locales.MN` GIVES IT AN ENTRY. Decided in coordination on 2026-11-12, over an order declared for every name (the pack's first draft, which misread the romanized names Mongolians write given-first, 65 of 87 two-word ORCID credits) and over no pack. The rule's license for a DEFAULT entry is unchanged: a script whose own convention settles the order, which Cyrillic, written for Russian given-first and Mongolian father's name first, is not; a pack's entry is a declaration, as a caller's own `script_orders` entry for katakana is (#594). What it took:
- `Script.CYRILLIC` and its spans (the Cyrillic block, Supplement and Extended-A, -B and -C) in `_policy._SCRIPT_RANGES`; the readers that mean the East Asian scripts read `_policy._EAST_ASIAN_SCRIPTS` instead of the whole table (rules/T3's interpunct flank and the two-family-names declaration's run count, `_classified_char`; the segmenter's neighbour test in `_script_segment`; the CJK corpus builder; the case table's CJK guard), so a parser built from `Policy` or the shipped packs without the pack reads every name as before (a caller's own `PolicyPatch` setting `script_orders` composes differently, below); `DEFAULT_SCRIPT_ORDERS` gains no entry; `_NO_INITIALS` gains none, Cyrillic names abbreviating.
- `_vocab._classify` asks the Cyrillic matcher only of a word opening with a Cyrillic letter, a C-level set test, so a non-ASCII Latin word costs no extra frame (the first build cost one a word, found by review; counted as `sys.setprofile` call events on 2026-11-12, `José García` reads 177 frames here as on master, and a Cyrillic name pays the matcher, `Иван Петров` 233 → 242); `_vocab.effective_script` classifies a Cyrillic word with a hyphen or an apostrophe inside it (`Оюун-Эрдэнэ`, `Мар'яна`) as Cyrillic, the joiners deleted from the classified copy only; the table stays letters only.
- `PolicyPatch.script_orders` composes BY SCRIPT instead of as a scalar: a patch's entry replaces the base's entry for its script and the base's other entries stand, so the pack adds its entry without erasing a caller's own opt-out (`Policy(script_orders=())` with MN keeps only the Cyrillic entry); `parser_for` warns where two packs set one script to different orders. An empty patch now clears nothing; a caller clears the table on `Policy` itself. This reaches a caller's own patch whether or not the pack is chosen (`Policy().patched(PolicyPatch(script_orders={Script.HAN: GIVEN_FIRST}))` keeps the Hangul and hiragana entries, where 2.1.0 through 3.0.1 kept the Han one alone), so the release log carries it as a Change; a malformed patch still reaches `Policy`'s messages, an unhashable key included.
- The entry orders a name and settles no lone word: `_assign`'s `EffectiveOrder.by_script`, which only rules/O5's lone-word sites read, is set for an East Asian script alone, so `Сүхээ` under MN reads given and reports `given-or-family` (W4's own scripts still decide a lone word, O5's W4 Except).
- Not carried by mn.toml's `[rules]`: the table's keys are scalar `Policy` fields, and a mapping keyed by `Script` would need a value type `tools/vocab/build.py`'s `RULE_KEYS` and `_locale._RULE_VALUES` do not have; the pack carries its patch in code, as ZH and JA carry `segment_scripts`.
- Accepted: a name not wholly Cyrillic keeps the declared order under the pack (a Latin title or suffix aside, the name words decide: `Dr. Батын Сүхээ` → family 'Батын', `Батын Sükhee` → given 'Батын'); and every Cyrillic name reads by the pack's order, Russian, Kazakh and Kyrgyz included, so `parser_for(locales.MN, locales.RU)` reads `Иван Сидоров` family 'Иван' and `Сидоров Иван Петрович` given 'Петрович', middle 'Иван', no patronymic (O1 stands down under a declared family-first order, and the patronymic lands in the given slot, where R9's East Slavic arm does not read), and `parser_for(locales.MN, locales.TR_AZ)` reads `Алиев Ильхам Гейдар оглы` given 'оглы'. Open: whether a patronymic shape the active rule reads (an O1 ending, an O2 marker) should keep its name from the pack's script order; no Mongolian name carries either, so the precedence would cost the pack nothing.

1 change: 1 addition & 0 deletions docs/design/decisions/deviates-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ Open: none in this file.
- 2026-10-30 #753, phase 3f — A CHOICE PACK MUST CHANGE A WORD. `locales.CS`, `SK`, `SL` and `HR` register as choice packs under phase 3d's contract, with no `DEVIATES`. `test_registry_is_the_pack_contract` adds one refusal: a choice pack whose choice changes no word (an empty `changed_words`), since its gate would pass on nothing; and `test_choice_rotators_cover_every_changed_word` holds each choice pack's rotator list to a name per changed word, so the gate's positive side reaches every word. The combined gate reads the surface of the union of every registered pack's languages, since `parser_for` applies one choice, not one per pack, or'd with every pack's hand `DEVIATES` whatever the pack, since a mixed pack carries both. decisions/vocabulary-model's 2026-10-30 phase 3f entry.
- 2026-10-30 #753, phase 3f amended — A CHOICE PACK THAT GAINS A PATCH DECLARES THE PATCH'S PART, AND THE PACKS SHARING A PATCH SHARE ONE DECLARATION. `locales.CS` and `locales.SK` gain rules/O6's patch (`two_family_names`), so they leave the choice packs and ship `DEVIATES`, as `locales.ES` does, the contract test requiring it of every pack that is not a choice alone. Their declaration is the one `ES` already made for the same patch: three or more name words, or three or more runs of letters. It moved from `locales/es.py` into `_locale.deviates_by_two_family_names`, which all three packs' `DEVIATES` return, so the three cannot drift apart (`test_cs_and_sk_pack_contents` holds them to it). Each pack's title and generation rotators are names the patch alone leaves as the default reads them (`MUDr Novák`, `Novák PhDr.`, `Jana Nováková (roz. Svobodová)`), since `test_a_choice_word_rotator_is_moved_by_the_choice_alone` refuses one the patch moves; the patch has rotators of its own. decisions/vocabulary-model's entry of this date.
- 2026-11-05 #1087 — THE TWO-FAMILY-NAMES DECLARATION COUNTS BY THE SCRIPT TABLE. `_locale._letter_runs` counted a run twice when its first character's Unicode name began `CJK`, `HANGUL`, `HIRAGANA` or `KATAKANA`, a second definition of East Asian text beside `_policy._SCRIPT_RANGES`, which every script-conditional rule reads. It now asks `_script_matcher` over the table, so a run W1 can divide is one the declaration counts twice. Moved first, before the table held the conjoining jamo, the declaration would have dropped decomposed `김. 민준` (#1083's refused candidate, which measured 4 of 6,731 names moving); with HANGUL widened (decisions/nfd-shapes, same date), 3 of the 6,738 distinct corpus texts moved (measured 2026-11-05, every distinct name `tools/differential/snapshot.py` reads, the comparator the character-name test run beside the table in one script), each gaining a declaration, the safe direction: `ヤマダ タロウ`, `タナカ. John` and `タロウ·ヤマダ`, halfwidth katakana the name test did not count. None lost one. Over all of Unicode (version 14, Python 3.11), the letters the name test counted and the table does not are the compatibility jamo, the katakana phonetic extensions, and the supplementary plane's Minnan tone kana and archaic and small kana, which no rule classifies and W1 never divides, so a run of them counts once correctly; the table adds 々, 〆 and the halfwidth katakana. `test_the_two_family_names_declaration_counts_by_the_script_table` in tests/v2/test_locales.py holds the rows with both former definitions' answers as recorded controls.
- 2026-11-12 #1237 — A CHOICE THAT CHANGES NO WORD, BESIDE A PATCH, IS RECORDED AS INERT. `locales.MN` chooses `mn`, whose readings are all on by default, beside a Cyrillic script order its patch carries in code (decisions/W4's 2026-11-12 #1237 entry), so its derived surface is empty. The phase 3f refusal is unchanged, since it reads only a pack that is nothing but its choice, and MN declares `DEVIATES` for its patch, which the gate reads; `test_choice_rotators_cover_every_changed_word` and `test_a_choice_word_rotator_is_moved_by_the_choice_alone` skip a pack listed in tests/v2/test_locales.py's `_INERT_CHOICES`, and `test_an_inert_choice_changes_no_word` fails the day such a choice changes a word, so the pack leaves the set and its rotators name the word. decisions/vocabulary-model's 2026-11-12 #1237 entry.
Loading
Loading