Skip to content

Read script properties from one private table: Georgian has no case, Burmese and Thai write no initials (#1256) - #1377

Merged
derek73 merged 16 commits into
masterfrom
feat/issue-1256-script-properties
Nov 13, 2026
Merged

derek73 merged 16 commits into
masterfrom
feat/issue-1256-script-properties

Conversation

@derek73

@derek73 derek73 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Implements the decision on #1256: one private table of script properties, read by every rule that asks, with the two leaves it settles.

Closes #1222
Closes #1234
Part of #1256

"Part of" rather than "Closes": three of #1256's sub-issues stay open after this PR.

decisions/script-properties lists all three under Open:. #1214 (the Cyrillic apostrophe) and #1224 (a particle opening a hyphenated word) are not rows: they are questions of a language's spelling, which no script column could hold, so they stay in decisions/R4.

What changed

  • The table (_policy._SCRIPT_PROPERTIES). Each row is keyed by a private name, never by Script, and has spans, has_case, writes_initials, period_abbreviations, spells_initials, a reason and a source. It replaces _policy._NO_INITIALS, _vocab._INDIC_BRAHMIC and Python's own case properties at every name-level case contrast. Readers build matchers once at import from the spans a column selects, so no reader adds a frame. The first commit moves no reading: snapshot.jsonl is byte-identical, and the frame band and serial benchmarks pass.

  • The split. _NO_INITIALS answered two questions: "writes initials" (T5, R7, the name match) and "writes period abbreviations" (H2's shape, S3's split and joined shapes). Thai answers them differently, so each is now its own column. One reader, period_joined_vocab (S3's dotted credential), was asking the wrong one. It moved in its own commit, with no reading moved.

  • Georgian (Should Georgian script count as caseless? (ჯეკ MA gives family '' and suffix 'MA', where محمد MA gives family 'MA') #1222). Georgian, Mkhedruli and Mtavruli, has no case for the name-case contrast. Six readers drop its letters with str.translate:

    • the one-case fact
    • the credential lean
    • R5's gate
    • S8's two name-word-behind predicates
    • R4's Mtavruli repair, which now reads the row and changes the same 46 letters over every codepoint

    T7's and R3's lower-case run of initials read the row in one commit and went back to Unicode after the design-docs review. That run asks whether a dotted run of single letters is initials, which Georgian answers yes in its one case, so ნ.ბ. ბერიძე reads given ნ., middle ბ. as on master.
    Both of the following leaves rest on #1256's amendment of 2026-11-12. Each is its own commit.

  • Burmese (Should initials() skip Burmese script, as it skips Han and Hangul? (ကြည် ကြည် ဌေး gives 'က. က. ဌ.') #1234). The Myanmar script writes no initials, decided from the writing convention. CLDR's my block is en's: all 42 patterns are equal once spaces and commas are removed, and it holds {surname-prefix} though Burmese names have no surname. (72c6766)

  • Thai. Thai writes no initials, decided by the CLDR count: th uses an initial in 0 of 20 locale-authored patterns. It does write period abbreviations, so ดร., ร.ต.อ. and พ.บ. read as before. The accepted cost is a pen name like ส. ศรีสุข: it keeps its fields, but initials() gives ''. (28f73e9)

What moved (snapshot, master 161ebd2 → head)

change parse fields initials() / capitalized() reports
table none none none
Georgian ჯეკ MA family MA; ჯეკ, MA given MA, family ჯეკ, as محمد, MA reads (both the 1.4.0–2.3.0 readings, checked against the wheels); ჯეკ ბერიძე Jr. ნინო family ნინო Ნინო Ბურჯანაძე → ნინო ბურჯანაძე (the one pre-existing snapshot entry that moves; the rule examples add more) suffix-or-name added where the count now decides
Burmese none by default; the same two policy moves as Thai (ဌေး က. ခ.) 39 entries, 35 distinct names, all CLDR fictional renderings order dropped before a Burmese letter (ကြည် က.)
Thai none by default; under initials_as_family ใจดี ส. ก. keeps middle ส., and under two_family_names สมชาย ส. ใจดี reads family ส. ใจดี (rules O8 and O6, Accepted) 73 entries, 71 distinct names, 20 of them CLDR-only order dropped before a Thai or Latin letter (สมชาย พ.ต.ส., สมชาย V)

Local corpora, snapshot.py --local … --base master (2026-11-12):

  • jrc-names: 565 spellings moved, 561 of them in initials() only (548 Thai, 13 Burmese). The other four are Georgian and match their Arabic twins on master: ალექსი II მოსკოვის პატრიარქი and კარლ XVI შვედეთი now capitalize the numeral as Ii and Xvi, and პეტრე I რუსეთი and ფრანსუა I საფრანგეთი now report conjunction-or-initial (see Accepted)
  • nepal-candidates: matches the base.
  • opensanctions-parliaments: matches the base.

A test from #1223 moved, deliberately. test_georgian_in_mtavruli_repairs_to_mkhedruli pinned Ნინო Ბურჯანაძე (a Mtavruli first letter on Mkhedruli) as kept on the default capitalized() path. With R5's gate reading the row, Georgian is one case to the gate, so the name is written in Mkhedruli, as force=True already wrote it. A Mtavruli surname beside a Mkhedruli given name (ნინო ᲑᲣᲠᲯᲐᲜᲐᲫᲔ) moves the same way. That differs from Latin, where Jean DUPONT is kept: Latin's capitals are case, while Mtavruli is a style of a caseless script. This is recorded as DECIDED in decisions/R4 and as an Accepted bullet in rules/R5. The test keeps a row for each shape with the new reading.

Recorded as Accepted

  • Word-level tests stay on Unicode: a Mtavruli word is a word in capitals. O9 still reads ᲑᲣᲠᲯᲐᲜᲐᲫᲔ ნინო under capitals_as_family as family ᲑᲣᲠᲯᲐᲜᲐᲫᲔ, so it loses no reading. Under that switch ჯეკ ბერიძე MA keeps family MA, unchanged from master.

  • A mid-name roman numeral in a Georgian name now repairs as it already did in an Arabic one (ალექსი II … → Ii, as محمد II علي حسن gives on master). That is a defect of caseless names in general, not of this row.

  • The Myanmar row's period_abbreviations stays true, there being no evidence either way: ကက. ဌေး still reads title ကက..

  • A Georgian dotted run followed by a credential that S8 reads as a name word stays whole. The credential's Latin capitals then count among the name words, which turns off T7's all-lower-case reading. So ნ.ბ. ბერიძე Jr. ნინო gives given ნ.ბ., middle ბერიძე Jr., family ნინო, where master gave given ნ., middle ბ., suffix Jr. ნინო. ნ.ბ. MA keeps its run whole. Master already reads a run whole beside a Hebrew or Arabic word (ნ.ბ. בן ბერიძე). Recorded in rules/T7's Accepted bullet.

  • A listed Georgian salutation behind a credential is the family name: ჯეკ ბერიძე MA ბატონი gives family ბატონი (master: title ბატონი, suffix MA); ქალბატონი and ქ-ნი read the same way. Its Hebrew and Latin Title-case twins already read this way on master (משה כהן MA מר, Jack Smith MA Mr), and behind Jr. it stays the title. Recorded in rules/S8's Accepted bullet.

Docs

  • rules/S6, R5 (an Except bullet and an Accepted bullet), R7 and H2's Accepted clause.
  • A new decisions/script-properties, with each row's evidence, the CLDR recipe with its copy check, and a dated table. Entries in decisions/R4, R5, R7, S6, H2, cjk-full-stops and initials-repertoire.
  • mechanisms.md#SCRIPT-PROPERTY-TABLE.
  • nameparser/AGENTS.md: a sweep for a split column.
  • Release-log bullets for Georgian (with the capitalized() move), Burmese and Thai.

Swept against master: every comma shape, the three orders, every policy switch, every locale pack, NFC and NFD, Parser.revise, both sides of matching().check, and Parser.suggest. 11 mutations of the changed conditions are each caught by a test, and so are the code review's row-value mutations.

Moves under the packs and the suggester, all recorded in decisions/script-properties:

  • Under the two-family-names packs (locales.CS, SK, ES, ES_AR), a Thai or Burmese letter with a period between two names is now the first family word rather than a middle initial: Nino ก. Beridze gives family ก. Beridze, as John 王. Smith already did on master. This goes through rules/O6.
  • Under locales.NE (initials_as_family), such letters are no run, through rules/O8.
  • A lone က. or ก. now reports given-or-family, as any lone word does.
  • A Burmese or Thai word before a Latin letter drops the order report (က V, สมชาย V).
  • Parser.suggest no longer suggests FAMILY_FIRST for a group of Thai or Burmese names, each a word followed by a letter. That is recorded in decisions/group-suggestion. suggest_measure.py --base master: 0 cells moved.

Commit e8e200b moved tags. It is "no reading moved" against master, not against its parent: 72c6766 had dropped shape:acronym and vocab:suffix-ambiguous from က.က. in က.က. က, and e8e200b gives them back. No field moved.

Mutations, on the final head (bb606fd)

Each changed condition was mutated alone, and the test-file set below was run against it (test_script_properties, test_render, test_match, pipeline/, test_cases, test_rules_doc). All 22 are killed:

mutation killed by
T7 run onto the table test_script_properties::test_a_georgian_run_of_initials_reads_as_initials
R3 run onto the table test_script_properties::test_a_georgian_run_ending_the_name_gives_an_initial_per_letter
S8 name behind off the table test_script_properties::test_georgian_reads_as_its_arabic_twin
S8 generation off the table test_script_properties::test_georgian_reads_as_its_arabic_twin
one-case fact off the table test_script_properties::test_has_case_reaches_every_contrast[georgian]
credential lean off the table test_script_properties::test_has_case_reaches_every_contrast[georgian]
R5 gate off the table test_script_properties::test_has_case_reaches_every_contrast[georgian]
R4 repair off the table test_script_properties::test_has_case_reaches_every_contrast[georgian]
S3 joined shape on the initials column test_script_properties::test_the_title_shape_reads_its_two_columns[myanmar]
S3 split pair on the initials column pipeline/test_vocab::test_two_dotted_letters_merge_only_outside_a_cased_script
H2 leading title on the initials column test_script_properties::test_a_script_with_no_initials_may_still_write_abbreviations (กข.)
_read_spans does not raise test_script_properties::test_a_column_with_no_row_fails_the_import_by_name
dict for the table's MappingProxyType test_script_properties::test_the_table_and_the_caseless_map_are_read_only
dict for the caseless map's MappingProxyType test_script_properties::test_the_table_and_the_caseless_map_are_read_only
Mtavruli range shrunk to U+1C90–1CA5 test_script_properties::test_every_mtavruli_letter_repairs_to_mkhedruli
brahmic period_abbreviations False test_script_properties::test_each_row_is_pinned_by_a_reading (राम शर्मा क.ख.)
myanmar period_abbreviations False test_script_properties::test_each_row_is_pinned_by_a_reading (ကက.)
myanmar Extended-A dropped test_script_properties::test_every_letter_of_a_row_script_is_in_the_row[myanmar]
myanmar Extended-B dropped test_script_properties::test_every_letter_of_a_row_script_is_in_the_row[myanmar]
myanmar main block halved test_script_properties::test_every_letter_of_a_row_script_is_in_the_row[myanmar]
thai row writes initials test_script_properties::test_a_script_with_no_initials_may_still_write_abbreviations (ดร.)
georgian row has case test_script_properties (the case-column tests)

Myanmar Extended-C (U+116D0–116FF) is the one span no test pins: it holds only digits as of Unicode 17. It stays in the row so that a letter assigned there later reads like the script's other letters, and it moves no reading today.

Checks

  • uv run pytest, -n0 -m serial, mypy, ruff, sphinx-build -b doctest (874, 0 failures), python -m doctest README.rst, tools/vocab/build.py --check
  • the suite on Python 3.15 in its own environment
  • the suite without namedivider-python, and with it

Review aspects run

  • code (pr-review-toolkit:code-reviewer): two findings, fixed in 1226317
  • comments (comment-analyzer): eight findings, fixed in b212787
  • design docs (design-docs-reviewer, all 12 axes): four findings, fixed in 1226317
  • independent docs review (from the coordinator): A1–A3 and B1–B13, fixed in b9db338. This includes reverting the T7 and R3 run sites, which had lost a reading
  • independent code review (from the coordinator): C1–C8, fixed in 3720c4a. C1 was already fixed by b9db338's revert, so no Arabic reading moves. The other fixes:
    • packs and the suggester swept and recorded
    • the table and the caseless map are read-only
    • has_case is typed Literal[False] | None
    • an empty column fails the import by name
    • row values and spans are pinned by readings and by Unicode letter names
    • the Script enum is bound to the table by span
  • tests (pr-test-analyzer, with 9 single-site mutations): six findings, fixed in 840528e. These are a widened source scan with a planted control, R4's repair added to the per-row test, Thai guards for is_leading_title and split_pair, the Georgian comma shape, and an in-suite negative control (11 twin rows apart with the caseless table emptied)

🤖 Generated with Claude Code

derek73 and others added 11 commits November 12, 2026 21:51
_policy._SCRIPT_PROPERTIES holds one row per script, keyed by a private
name rather than by Script: whether the name-case contrast reads case in
it, whether it writes initials, whether it writes period abbreviations,
and whether a bare run of its letters with a period is a spelled Latin
initial, each row with its reason and source. The readers that asked
_NO_INITIALS, _INDIC_BRAHMIC and Python's case properties at the three
contrast sites now read spans selected from it, built once at import.

"Writes initials" (T5, R7, the name match) and "writes period
abbreviations" (H2's shape, S3's joined Ph. D.) are separate columns,
which the four East Asian rows answer alike. No reading moves:
snapshot.jsonl byte-identical, the frame band unmoved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Python counts every Mkhedruli letter as lower case, so a Georgian name
beside a Latin capital was mixed case and the capitals leaned to the
credential: 'ჯეკ MA' read suffix 'MA' and no family from 2.4.0 through
3.0.1. The script-property table's Georgian row (Mkhedruli and
Mtavruli) has no case for the contrast, so the one-case fact, the
credential lean and the case-repair gate drop its letters, and the name
reads as 'محمد MA' does: family 'MA', with a suffix-or-name report.

rules/R4's Mtavruli repair reads the same row, changing the same 46
letters over every codepoint. capitalized() now repairs a Mtavruli word
or first letter beside Mkhedruli on the default path, the snapshot's
one move ('Უჩა გოგოლაძე' -> 'უჩა გოგოლაძე').

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Burmese abbreviates no name to a letter and a period, and the first
character of a syllable with a medial is not even its onset as written:
'ကြည် ကြည် ဌေး'.initials() gave 'က. က. ဌ.', ကြည် (Kyi) and က (Kyaw)
alike. The script-property table's Myanmar row writes no initials, so
initials() gives '' for a Burmese word, as for Han, Hangul and kana, the
name match reads a Burmese letter as a whole name, and T5 tags none as
an initial. No parse field moves; 39 snapshot entries move initials()
only, 35 of them CLDR's fictional renderings.

The row rests on the convention, decided on #1234, not on CLDR: my's
personNames block is en's copied (36 of 42 patterns identical once the
comma is written ၊, the rest differing by a space), so the recipe does
not answer. Whether the script writes period abbreviations is left as
it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…column (#1256)

period_joined_vocab's shape verdict (rules/S3) refuses a script with no
period abbreviations, as its docstring says, but asked the initials
column. The two columns agree for every row so far, so nothing moves;
they part for Thai, whose degrees are written as dotted consonant runs
(พ.บ.).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns (#1256)

CLDR's th authors 20 name patterns and uses an initial in none, its own
block rather than en's, so Thai joins the scripts that write no
initials: 'สมชาย ใจดี'.initials() gives '' where 1.4.0 through 3.0.1
gave 'ส. ใ.', and the name match reads a Thai letter as a whole name.
Thai still writes titles and degrees short with periods (ดร., ร.ต.อ.,
พ.บ.), so the row keeps period abbreviations and those read as before.

73 snapshot entries move, 72 in initials() only; 'สมชาย พ.ต.ส.' also
drops its order report, which reads a word in a script that writes
initials, as a byline's, its reading unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1256)

mechanisms.md gains SCRIPT-PROPERTY-TABLE; rules/H2's Accepted clause
states its period-abbreviation veto as its own question rather than the
initials veto's; nameparser/AGENTS.md records the sweep a split column
asks for, with the S3 reader the Thai row found on the wrong column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decisions/R4 records the default-path move as decided, naming both
shapes with their readings in 1.4.0 through 3.0.1 (kept as written) and
how it differs from a Latin surname in capitals; rules/R5 gains it as an
Accepted bullet; the release-log bullet names the capitalized() move.
#1223's test pinning the Latin-habit capital as kept moves with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the comment review: the FULL_STOPS bundle's fifth site and the
period-abbreviation veto name the column they read and all four of its
readers, the ASCII shortcut's bound is U+007F (the conjoining jamo start
at U+1100), the table header claims only its four questions, the
caseless-letter table names R4's reader, and Cyrillic is said to have
no row rather than one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the code review: the lower-case run of initials (rules/T7 in
_assemble, R3 in initials()) and S8's name word behind a credential or
generation word asked Python's case, so a Georgian word counted as lower
case there. They drop the caseless rows' letters as the other contrast
sites do: 'j.r. ბერიძე' reads given 'j.r.' and 'ჯეკ ბერიძე Jr. ნინო'
middle 'ბერიძე Jr.', each as its Arabic twin. A twin test holds fifteen
shapes, ten of which master failed. The design-docs review's fixes:
stale Standing lines on H2's veto and the scripts with no initials,
the test's reach stated as it is, snapshot counts by distinct name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The source scan now refuses any character range overlapping a row
(escapes, hex pairs, chained comparisons, literal characters in a
class), with a planted control of what it sees and lets pass and a
named allowlist for the Thai marks other rules read. R4's repair joins
the per-row case test; Thai rows guard is_leading_title and split_pair
on the abbreviation column; the Georgian comma shape joins the twin
test, whose recorded negative control (11 rows apart with the caseless
table emptied) runs in the suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 3.1 milestone Nov 13, 2026
@derek73 derek73 added design Changes what the model can express (a shape decision) non-latin-script Names written in a script other than Latin (rules/W) initials Reading initials and the initials() view (rules/T) case-repair capitalized() case repair (rules/R) internals Refactors and parse cost with no behavior change labels Nov 13, 2026
@derek73 derek73 self-assigned this Nov 13, 2026
derek73 and others added 5 commits November 12, 2026 23:09
codecov/patch found _ranges_pattern's refusal of an empty selection
unrun; a test now selects a value no row holds. _capitalize's guard is
one condition, its longer-lower-case arm having no codepoint to reach.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds corrected (#1256)

A1: rules/T7's and R3's lower-case run of initials go back to Unicode's
case. They ask whether a dotted run of single letters is initials, which
Georgian answers yes in its one case, so 'ნ.ბ. ბერიძე' reads given 'ნ.',
middle 'ბ.' and 'j.r. ბერიძე' given 'j.', middle 'r.', as on master; the
twin test, an oracle only where the scripts differ in the column under
test, now holds sixteen shapes (eleven apart on master, a recorded
control) and the dotted runs have tests of their own. S8's two sites
keep reading the table: a gain, by S8's own caseless clause.

A2/A3: Thai and Burmese letters being no initials moves O8 under
initials_as_family and O6 under two_family_names, and drops the order
report before a Burmese letter or a Latin one behind Thai: recorded in
rules/O8 and O6 (Accepted, with examples) and the decisions entry.

B1-B13: one copy test for CLDR, stated and run (my 42 of 42); the
initialPattern step restored to the recipe; the Burmese row's source is
the 2026-11-12 amendment to #1256's decision; #1214 and #1224 are not
rows; what a script with no row reads as, and what was not examined;
rows' reasons stated as conventions with dated sources; R5's Except
narrowed to its boundary; the snapshot's one moving entry named;
Standing bullets moved into their lists; release-log bullets split and
reworded; migrate.rst's 3.1 changes; nameparser/AGENTS.md's lesson
completed; stale _NO_INITIALS mentions in tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ows pinned (#1256)

C2/C3: every locale pack and Policy switch swept over the Thai, Burmese
and Georgian grids; the two-family-names packs, locales.NE, a lone
letter's report and the order report before a Latin letter recorded in
decisions/script-properties, and Parser.suggest's trailing-initial vote
in decisions/group-suggestion (suggest_measure --base master: 0 cells
moved). C4: S3's paired initials in a script with capitals recorded as a
site left on Unicode by decision. C5: e8e200b restored two tags
72c6766 had dropped, recorded. The Burmese and Thai rows point at
#1256's 2026-11-12 amendment, with Thai's pen-name cost.

C6: each row's values pinned by a reading, its spans by the Unicode names
of its script's letters (U+A8FE, Devanagari Extended, named outside the
brahmic row), all 46 Mtavruli letters through R4. C7: the table and the
caseless map read-only, has_case typed Literal[False] | None, an empty
column failing the import by name, the guard-off control patching each
reader's binding. C8: every Script member's spans in a row or a stated
default (Cyrillic), with its control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipt-properties

# Conflicts:
#	docs/design/decisions/group-suggestion.md
#	tests/v2/test_cases.py
…1256)

Records and tests only; no reading moves. rules/T7 and S8 gain Accepted
bullets, with examples, for the two unwritten shapes the independent
review found: a Georgian dotted run left whole once S8 reads a Latin
credential behind it as a name word (as beside a Hebrew word on master),
and a listed Georgian salutation behind a credential read as the family
(as its Hebrew and Latin Title-case twins read). Tests assert both
script-table mappings are read-only, give R3's Georgian run its own row,
and say why Myanmar Extended-C, digits only, pins no letter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 82d66ad into master Nov 13, 2026
14 checks passed
@derek73
derek73 deleted the feat/issue-1256-script-properties branch November 13, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

case-repair capitalized() case repair (rules/R) design Changes what the model can express (a shape decision) initials Reading initials and the initials() view (rules/T) internals Refactors and parse cost with no behavior change non-latin-script Names written in a script other than Latin (rules/W)

Projects

None yet

1 participant