Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions docs/design/decisions/W1-unspaced-cjk-division.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@
- Script classification composes its input to NFC while segmentation reads the raw text, so a decomposed name degrades to no split, never a wrong one: `김민준` typed decomposed reads family '김민준'; vocabulary matching composes too since #322, the correction inside the entry — 2026-07-29.
- `locales.ZH` and `locales.JA` are corpus alternatives; stacked, a listed Chinese surname still wins before the segmenter (`高橋一郎` divides 高 + 橋一郎 under ZH+JA as under ZH alone), and kana gating follows the script function order uses, so a wholly-katakana token never gates in — 2026-08.
- The ZH single-character list is the census rows plus the union of Taiwan's and Wikidata's top 300, written by `tools/zh_surnames.py`, so `柯洁` and `关羽` under `locales.ZH` read family '柯' and '关' — 2026-10-18 #695.
- Accepted under `locales.ZH`: a Japanese name splits wrongly more often (`安達禎` → family '安'), and so does a transcription written without the 间隔号 (`安娜` → family '安', given '娜') — 2026-10-18 #695.
- Five mainland surnames that both rankings miss are added by hand at the census block's tail, with the traditional forms Taiwan attests, each annotated with its mainland rank, so `付晓明` under `locales.ZH` reads family '付', given '晓明'; a mainland-weighted third source is the Open line — 2026-10-30 #952.
- Accepted under `locales.ZH`: a Japanese name splits wrongly more often (`安達禎` → family '安', and since #952 `代田稔` → family '代' and `芦川誠` → family '芦', 芦 opening 102 of a list of 172,413 Japanese names), and so does a transcription written without the 间隔号 (`安娜` → family '安', given '娜') — 2026-10-18 #695, amended 2026-10-30 #952.
- A listed compound wins over its first character whatever the name's length, and a token that is wholly a surname stays whole: `张简明` → family '张简', given '明', and `张简` → family '张简' under `locales.ZH`; compounds stay hand-curated against bearer data, 东野 and 成公 left out — 2026-10-18 #695.
- A pack holding every ranked surname, a stricter divide-after-the-first-character mode, the unlicensed or copyrighted sources and the unattested Hundred Family Surnames compounds are declined — 2026-10-18 #695.
- A name the interpunct divides stands division down, vocabulary and segmenter alike (decisions/T3, #298), and rules/W1 states it as an Except rather than an Accepted cost: `安东尼·陈志明` under `locales.ZH` → family '陈志明', given '安东尼' — 2026-10-20 (Derek), #761.

Elsewhere: a segmenter is asked only of a name its writer left undivided, decisions/W3's 2026-08 entry; the Japanese surnames and married doubles kept out of the ZH list, decisions/vocabulary-exclusions' #695 lines.

Open: none in this file.
Open: a mainland-weighted ranking as a third source in `tools/zh_surnames.py`, which would catch the next surname both rankings miss without a hand addition — the route the 2026-10-30 #952 entry did not take; wainshine/Chinese-Names-Corpus, the candidate it weighed, is under Sources not used as data.

## Log

Expand All @@ -32,13 +33,16 @@ Open: none in this file.
Accepted: a listed compound wins over its first character in every name that opens with it, whatever the name's length — longest match first, the same mechanism the married-double exclusion in decisions/vocabulary-exclusions guards against. 张简明 reads 张简 + 明 where it had read 张 + 简明, 贺兰芳 reads 贺兰 + 芳, 范姜伟 reads 范姜 + 伟, and a two-character name that spells a compound stays whole (张简 is family 张简, not 张 + 简, a token that is wholly a surname not being divided). This was already true of the 34 shipped compounds (司马), and the compounds admitted are borne as one surname — 張簡 by 9,162 people in the Taiwan file — where a married double is two. The cost is on record: Wikidata has 25 people whose family name is the 张 item alone and whose Chinese label opens with 张简 (measured 2026-10-18). 14 are labelled with the two characters alone, 9 of them with given name 简 stated, so they now read whole where they had read 张 + 简; the longer ones are Taiwanese 張簡 bearers tagged with the single character or undetermined (the Tang and Ming 张简修). Recompute: people with P734 = Q804909 and a zh, zh-hans, zh-hant, zh-cn or zh-tw label starting 张简 or 張簡, with their P735 labels.
The cutoff (wrong Japanese splits of 60, Chinese rows right of 80, each list run through the tool's `select` with the census rows and the shipped compounds): Wikidata alone at 200, 4 and 79; at 300, 9 and 80; at 400, 12 and 80. Taiwan alone at 200, 4 and 78; at 300, 10 and 79 (茅 is its 326th, in `read_moi`'s order); at 400, 12 and 80. Both at 200, 5 and 79; at 300, 11 and 80; at 400, 14 and 80. The corpus does not force 300: both lists at 212 read all 80 rows with 7 wrong Japanese splits, Wikidata alone at 212 with 6, and 211 misses a row. 300 leaves a margin over a corpus of 80 public figures, who skew toward common surnames (the reason #695 gave for expecting ordinary data to miss more), at the cost of 4 more Japanese wrong splits there than 212. (The first measurement, taken before the Unihan step existed, had Taiwan at 300 reading 77: without simplified forms it missed 卫 and 华.)
Compounds stay hand-curated (the note above `_COMPOUND` in zh.py states the admission test). A compound ships where Wikidata or the Taiwan file attests a bearer, and where no Wikidata bearer of its first character alone has a name opening with it; the check was read by hand, since most of its hits are compound bearers tagged with the single character (左丘明 under 左, 鍾離權 under 鍾). Amended the same day, after the second review: the check also fails where such a bearer's stated given name (P735) opens with the compound's second character, and a compound that fails it still ships when its living bearers in the Taiwan file outnumber those readings by orders of magnitude. 張簡 is the case (9,162 bearers against 9 people named 張 + 簡, the cost this entry's compound Accepted clause records); 东野 fails without it (no Taiwan bearers against 2 people named 东 + 野) and is left out. 25 simplified compounds were added, 钟离 and 段干 among them. Recompute the check: for each compound, people whose P734 is the item labelled with its first character, whose Chinese label opens with the compound, and whose P735 label opens with its second character.
- 2026-10-30 #952 (decision) — FIVE MAINLAND SURNAMES BOTH RANKINGS MISS ARE ADDED BY HAND, NOT BY A THIRD SOURCE. Five mainland surnames past both rankings' top 300 are added by hand at the tail of zh.py's census block, annotated there as 萧 is, rather than through a third source: 付, 代, 候, 芦 and 农, the mainland's 65th, 106th, 187th, 197th and 198th surnames by people in China's national ID registry (NCIIC, 2008, read from the CRAN package ChineseNames as a cross-check only), and with them 蘆 and 農, the traditional forms of 芦 and 农 that the Taiwan ranking attests (its 642nd and 317th), the condition `tools/zh_surnames.py` sets for shipping a traditional form. They are the unlisted surnames in the mainland top 200. Each ranking misses them through its regional bias: Taiwan ranks 付, 代 and 候 435th, 545th and 351st, since it writes 傅 where mainland registration writes 付 (候 looks like the same pattern for 侯), and Wikidata, as fetched 2026-10-18, ranks them 393rd, 589th and 610th and 农 585th (its 農 item, 10 bearers), with no item for 芦. The hand-kept rows sit before the generated block, so the tool reads them as listed and its block came out byte-identical. A third source was not taken (the Open line): its candidate, wainshine/Chinese-Names-Corpus, ranks distinct names extracted from text, not people.
Measured on 4ebb51d2 against this change, with `parser_for(locales.ZH)`: of the 1,144,226 names in wainshine/Chinese-Names-Corpus's mainland list, 34,472 were left whole before and 28,960 after, the 5,512 that moved being every name in the list that opens with one of the five (付 3,488, 代 1,323, 芦 371, 农 213, 候 117). Of the 308,667 names in the local `tw-donors` corpus (data date 2026-10-05), 21 moved, each from whole to divided (農 7, 候 6, 蘆 5, 付 3), of the 22 opening with one of the seven forms; the 22nd, 農太太1314號, holds digits, and one of the moves is not a name but a committee named for the district 蘆洲 (蘆洲區忠義國小家長委員會(楊建忠), now family '蘆'). The CLDR and Wikidata native locale views did not move, none of their names holding the seven forms, and the default snapshot moved none of the names it held; it gained this change's case rows, each read whole by default. Recompute: fetch `Chinese_Names_Corpus/Chinese_Names_Corpus(120W).txt` from that repository, take every non-empty line after its three header lines, parse each under `parser_for(locales.ZH)` at the parent and at this commit, and count a name whose family or given is empty; read the cached `tw-donors` `corpus.jsonl` the same way, counting the names whose reading differs.
Accepted: 代 and 芦 open Japanese surnames (代田, 芦田, 芦川, 芦名). Of the 172,413 names in the same repository's Japanese list, ZH divided 40,668 (23.6%) before and 40,805 after: 14 opening with 代 (代田稔 → family '代'), 102 with 芦 (芦川誠 → family '芦') and 21 with 蘆, the 芦 rows about 0.06 points of the list (0.07 with 蘆); 付, 候, 农 and 農 open none. 芦 cost the most of the five and was weighed on its own before it was added. Recompute: `Japanese_Names_Corpus/Japanese_Names_Corpus(18W).txt`, read as above, counting a name whose family and given are both non-empty (a name whose family is a proper prefix of it gives 40,673 and 40,810, the five more being the honorific peel's, 八幡様 → family '八幡').
- 2026-10-20 (Derek), #761 — the interpunct stand-down moves from Accepted to Except in the skeleton migration (PR #782): a name the interpunct divides is inside When and Then stops holding there, which is the format's definition of an exception. Words and example unchanged.

Declined:

- JMnedict as bundled segmentation data (#272's body, 2026-07-07) — the pip packaging (jamdict-data) was last compiled 2021-04; JMnedict carries no frequency data, so it cannot resolve the ambiguous 2+2/3+1 splits that motivate a segmenter at all; and bundling raised licensing questions the core avoids by delegating to namedivider — whose code and GBDT model are MIT, whose surname data carries Myoji-Yurai terms permitting exactly this name-dividing use, and whose CC-BY-SA BERT model is unused (the 2026-07-29 amendment corrected an earlier flat-"MIT" description; so does this entry). The kind of decline someone re-proposes in two years.
- A pack with every surname either source ranks (#695 review, 2026-10-18). Taiwan's single-character surnames past the 300th hold 77,355 of its 23,373,283 people (0.33%). Recompute over the order `tools/zh_surnames.py`'s `read_moi` returns — single unified ideographs ranked among themselves, which is not the CSV's `ranking` column, since that column counts two-character surnames too — summing the population column for every surname past index 300. The tail is where the characters that open Japanese surnames sit (in that order: 山 369, 川 656, 大 787, 小 788, 中 920, 本 1008). For data that is Chinese, a complete list acts as "divide after the first character unless a compound opens the name"; the list's work past the head is screening out text that is not a Chinese name, and the tail stops paying for it. A stricter "divide after the first character" mode was raised in the same review and declined: one ZH pack.
- Sources not used as data: the PRC Ministry of Public Security name reports (top 100 only, no licence); Yuan Yida's 300- and 400-surname books (copyrighted; usable as a cross-check only); the CRAN package ChineseNames (GPL-3, not redistributable here); zh.wikipedia's 中国姓氏排名 tables (CC BY-SA, compiled from the sources above).
- Sources not used as data: the PRC Ministry of Public Security name reports (top 100 only, no licence); Yuan Yida's 300- and 400-surname books (copyrighted; usable as a cross-check only); the CRAN package ChineseNames (GPL-3, not redistributable here; #952 read its registry counts as a cross-check); wainshine/Chinese-Names-Corpus (Apache-2.0, but a list of distinct names a word splitter extracted from text, with no counts of people and some bad entries by its own README, so its ranking of first characters measures how often a surname occurs in text rather than how many people bear it; #952 measured with it); zh.wikipedia's 中国姓氏排名 tables (CC BY-SA, compiled from the sources above).
- Hundred Family Surnames compounds no source attests a bearer of (宗政, 谷梁, 东门, 南门, 归海 and others). They divide nothing real, and where the first character is a listed surname they would cut a name that opens with it (宗 + 政X).
- 东野 — two Wikidata people named 东野 have 野 as their stated given name, and the Taiwan file has no 東野 bearer to outweigh them (the amended compound test, 2026-10-18).
- 成公 — 成公亮, the guqin player, is 成 + 公亮, and a Wikidata bearer of 成 whose name opens with 成公 is the case the compound test exists to refuse.
1 change: 1 addition & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Release Log
- **Render a name in the order it was read in, keeping its written separators.** ``str(parse("김민준"))`` gives ``'김민준'``, where 2.1.0 through 2.8.0 gave ``'민준 김'``; a name read family-first renders family-first, and a comma before a suffix stays (``John Smith, Jr.``, ``Smith, Jr.``), while a family comma between name words never does (``Smith, John`` gives ``'John Smith'``). A spec you pass to ``render()`` renders as written, so 2.x's default spec keeps given-first output, and a name pickled by 2.x renders as it did. ``initials()`` gives nothing for a word in Han, Hangul or kana. The new ``ParsedName.order`` records the order; ``render()``'s spec now defaults to ``None``. See :doc:`migrate`. :issue:`754`
- **Stop warning when two locale packs set a policy field to the same value.** ``parser_for(locales.CS, locales.SK)``, which both turn ``two_family_names`` on, is silent, as is any pair of packs agreeing on a value, however each spells it, where 2.0.0 through 2.8.0 warned that the later pack overrides the field. Packs setting different values still warn, the later winning. :issue:`753`
- **Fix Policy(name_order=...) accepting a set.** ``Policy(name_order=frozenset(FAMILY_FIRST))`` raises ``TypeError``, where 2.0.0 through 2.8.0 accepted it as an order under some hash seeds and raised ``ValueError`` under others; pass the tuple. :issue:`754`
- **Divide five more mainland Chinese surnames under locales.ZH.** ``付晓明`` under ``locales.ZH`` gives family ``付``, given ``晓明``, where 2.1.0 through 2.8.0 gave family ``付晓明``; ``代``, ``候``, ``芦`` (``蘆``) and ``农`` (``農``) divide too. A Japanese name opening with ``代`` or ``芦`` now splits after it (``芦川誠`` gives family ``芦``). :issue:`952`

* 2.8.0 - October 28, 2026

Expand Down
22 changes: 18 additions & 4 deletions nameparser/locales/zh.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@
Ministry of the Interior surname ranking (內政部戶政司, 2023, used under
the Open Government Data License, version 1) and Wikidata's Chinese
family names ranked by bearers (CC0). tools/zh_surnames.py derives
that block and writes it below; do not edit it by hand.
that block and writes it below; do not edit it by hand. Five mainland
surnames that neither ranking reaches are added by hand at the census
block's tail (#952), each annotated there with its mainland rank.
Traditional-script variant forms are included for names written in
Taiwan/Hong Kong/overseas orthography; two-character compounds are
hand-curated (see the note above _COMPOUND), never generated, since a
Expand Down Expand Up @@ -47,7 +49,7 @@
# 2020 census top-100 plus the annotated additions below, simplified
# forms, census rank order, 10 per row -- append new entries at the
# tail, do not alphabetize (rank order is the only in-file record of
# where the coverage floor sits). Two entries are NOT census rows and
# where the coverage floor sits). Seven entries are NOT census rows and
# are called out so the rank-order record stays honest:
#
# * 阎 sits off-rank beside 闫 rather than at the tail, because the two
Expand All @@ -58,6 +60,18 @@
# the merged 肖 (rank 33 above), but 萧 stays a distinct simplified
# spelling in current use and both circulate. The traditional form
# 蕭 ships either way, below.
# * 付, 代, 候, 芦 and 农 follow it (#952): mainland surnames ranked
# 65th, 106th, 187th, 197th and 198th by people in China's national ID
# registry (NCIIC, 2008), the unlisted ones in its top 200. Both
# rankings miss them for their regional bias: the 2023 Taiwan file
# ranks them 435th, 545th, 351st, 642nd (as 蘆) and 317th (as 農),
# and Wikidata, as fetched 2026-10-18, ranks all but 芦, which has no
# item, 393rd, 589th, 610th and 585th (as 農). 付 stands for 傅 in
# mainland registration and 候 looks like the same pattern for 侯;
# the 2023 Taiwan file has under 400 bearers of either. 蘆 and 農 are
# their traditional forms, below, the Taiwan ranking attesting each.
# 代 and 芦 open Japanese surnames (代田, 芦田), a cost decisions/W1
# records with the measurements (its 2026-10-30 #952 entry).
_SINGLE = (
"王", "李", "张", "刘", "陈", "杨", "黄", "赵", "吴", "周",
"徐", "孙", "马", "朱", "胡", "郭", "何", "林", "高", "罗",
Expand All @@ -69,7 +83,7 @@
"孟", "秦", "邱", "侯", "江", "尹", "薛", "闫", "阎", "段",
"雷", "龙", "黎", "史", "陶", "贺", "毛", "郝", "顾", "龚",
"邵", "万", "覃", "武", "钱", "戴", "严", "莫", "孔", "向",
"常", "萧",
"常", "萧", "付", "代", "候", "芦", "农",
)
# Traditional-script forms of the above where the glyph differs. 蕭 is
# the traditional spelling of both 萧 and the merged 肖 (see the note
Expand All @@ -78,7 +92,7 @@
"張", "劉", "陳", "楊", "黃", "趙", "吳", "孫", "馬", "羅",
"鄭", "謝", "許", "韓", "鄧", "馮", "蕭", "葉", "蘇", "呂",
"盧", "鍾", "譚", "陸", "韋", "賈", "鄒", "閆", "閻", "龍",
"賀", "顧", "龔", "萬", "錢", "嚴", "蔣",
"賀", "顧", "龔", "萬", "錢", "嚴", "蔣", "蘆", "農",
)
# Both rankings' top 300 past the census rows, in combined rank order
# (each surname's better rank of the two), both scripts.
Expand Down
Loading
Loading