Repository navigation
Read the fullwidth full stop between Han names as the transcription dot (#1312) - #1325
Merged
Merged
Conversation
… transcription dot (#1312) U+FF0E takes rule T3's flank guard and W4's transcription record, so 台邦．撒沙勒 divides as 台邦·撒沙勒 does; with a Latin neighbor it stays a plain full stop, and the ideographic full stop stays out. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
derek73
added
non-latin-script
…ulary word between East Asian characters (#1312) Review of the first commit found str() writing a space where the parse divided at U+FF0E, which reparsed as the opposite order, and the vocabulary build check refusing only U+00B7 inside a word the tokenizer now divides. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1312.
Taiwan writes Indigenous names in Han with a dot between given name and family name, and some writers type the fullwidth full stop U+FF0E where others type U+00B7. The parser read
台邦．撒沙勒as one given name with agiven-or-familyreport.Change
U+FF0E takes T3's (the interpunct's flank guard) rule whole: between two characters of a classified script (Han, Hangul, kana) it divides and records a transcription, so the name keeps its declared order (W4); with a Latin neighbor it stays a plain full stop.
台邦．撒沙勒台邦．撒沙勒台邦, family撒沙勒Kilang．Ｍanah台邦。撒沙勒Left out: the ideographic full stop U+3002, the sentence stop; no measurement of it between names was taken (said so in decisions/T3).
Known move:
田中．太郎now reads given-first like田中·太郎, not as a family-first roster pair (decisions/T3's codepoint scope).Found in review (second commit)
str()wrote a space where the parse divided at．, sostr(parse("台邦．撒沙勒"))reparsed in the opposite order;_render._DOT_SEPARATORSnow writes it back (R6 example, render row).tools/vocab/build.pyrefused only U+00B7 inside a vocabulary word between East Asian characters; it now refuses both (TRANSCRIPTION_DOTS, a mutation row, the sync pin).．in 15 distinct names (14 between Han characters), jrc-names in 2 (1 between Han), not the issue's 16 and once; the LatinPh。Dis U+3002. decisions/T3 and T8 corrected.Docs and records
Ph。Dcase.Verified
uv run pytest(43,933 passed), mypy, ruff. Not run: serial benchmarks, Python 3.15, tw-donors re-read against the base.🤖 Generated with Claude Code