Replace Unihan with Wikidata for locales.ZH's simplified forms and the suggester's pinyin - #1371

Based on master since #1370 merged (master merged in; the tree is unchanged from the reviewed commits).

What

No shipped data derives from Unihan any more, so Unicode-3.0 leaves the licence expression and LICENSE-unicode leaves the tree. This settles the Open item in #1370's FACTS WE MEASURE entry for Unicode-3.0. It does not settle #981 (a mainland-weighted third ranking). W1's Open line now notes that such a ranking would restore the 7 dropped forms.

• locales.ZH's simplified surname forms (tools/zh_surnames.py): a ranked traditional form gains the one simplified form given by the simplified-Chinese labels (zh-hans, zh-cn, zh-sg, zh-my) of the Wikidata items labelled with it (CC0). The items are every family name (Q101352) and every Chinese family name (Q1093580): 關, 顏 and 遲 have items of the first class only. It is the rule Unihan's variant table was read by: exactly one form, in the BMP, and not the form itself.
• The group suggester's pinyin surnames (tools/vocab/order_names.py): every Hanyu Pinyin reading that Wikidata's family-name items state for each locales.ZH single-character surname, read from three statements. They are P2440 qualified by P459 = Hanyu Pinyin (Q42222), the P1721 qualifier of P1705, and the P1705 of the Latin-script item that P460 names under P1013 = Q42222. A surname with no reading is left out, and the run names it: 35 of 463.
• Both tools key their Wikidata cache by the exact queries they send, ask in batches of 100, and refuse an answer that holds a row which is not a real row. That last case is how the query service's error text reads when it cuts a CSV answer short at its time limit; this PR's first run hit it.

Moves

locales.ZH. The block holds 317 forms, down from 324. 12 of the 19 forms only Unihan supplied are attested by Wikidata: 东云兰关区娄盘练迟阙颜风. The other 7 are not, and they leave: 买厐盖篮芈诸谌.

• I compared the two trees over every name that holds a Han character in the committed and local corpora: config_diff.py diff --committed --local all --match '[一-鿿㐀-䶿]' --readings <file> default plain:ZH, run in each tree, with the files compared line by line.
• 9 of 322,895 names moved. All 9 are transliterated foreign names that begin with 盖, such as 盖尔哈尔德 (Gerhard) and 盖达尔 阿利耶夫 (Heydar Aliyev). The pack used to cut them after 盖; now they read as the default does.
• The cost is real Chinese names with one of the 7 surnames written in simplified characters. 谌龙 (Chen Long) now reads undivided, as family 谌龙. Written in traditional characters (諶龍), the name still divides.
• cldr_agreement.py: no record moved, the CLDR and Wikidata-native locale views included.

Pinyin. The block holds 200 spellings, down from 206.

• Joined: zeng, shan, zha, zhai, bu, nian. These are the surname readings of 曾, 单, 查, 翟, 卜, 粘, where Unihan's first kMandarin reading gave ceng, dan, cha, di, bo, zhan.
• Left: an, bing, bo, ceng, cha, chou, dan, diao, jie, kou, nong, que. an stays in the table through the Korean list.
• The given-name table did not move.

Suggester. suggest_measure.py --base <e9cb3520>:

• 59 cells of the order table moved: 54 up and 5 down, each down by one group in 300.
• The largest move is PSC's Chinese filers written given-first under a family-first base: 59.7 / 84.3 / 91.7 / 94.7% became 63.0 / 85.7 / 92.3 / 96.0% at 2 to 5 names.
• No same-order cell fell, and the worst same-order row outside the three anti-goal rows still reads 98%.
• No population's first size at 95% moved, so the figures quoted in Parser.suggest, docs/suggest.rst and decisions/group-suggestion stand.

Docs

• decisions/shipped-data-licences: a 2026-11-12 UNIHAN REPLACED entry and a Standing line. It keeps the record that 2.5.0 through 3.0.1 shipped Unihan-derived forms.
• decisions/group-suggestion: a PINYIN FROM WIKIDATA entry, plus a head note on the TABLES entry.
• decisions/W1: a log entry, plus a head note on the 柯洁 reads as one family name under locales.ZH: should the Chinese surname list go past the census top 100? #695 entry.
• Also updated: AGENTS.md's "Shipping data" paragraph, README's License section, zh.py's and _order_names.py's provenance, and a release-log bullet.

Checks

• uv run pytest: 46,827 passed, 38 skipped, 3 xfailed.
• uv run pytest -n0 -m serial: 38 passed.
• uv run mypy and uv run ruff check: clean.

Reviews

• design-docs-reviewer on the docs/design changes. It found a cell count that counted listing lines, an missing, three anti-goal rows written as two, superseded entries without head notes, and AGENTS.md. All fixed.
• pr-review-toolkit code-reviewer. It found that the cache key ignored the query, the item that writes a form the same in both scripts, untested loaders, and the release-log wording. All fixed. Fixing it uncovered the cut-answer case.
• A second code-reviewer pass on the fix commit. It found that the order_names tests patched a copy of name_usage other than the one ask calls (in one collection order they reached the network), and that the pinyin cache test passed on the parent code. It also found that an empty pinyin batch was refused. All fixed in the third commit.

🤖 Generated with Claude Code