Skip to content

Drop nam_dict's familiar forms, keeping the nicknames list's - #1372

Merged
derek73 merged 2 commits into
masterfrom
feat/drop-nam-dict-familiar-forms
Nov 13, 2026
Merged

derek73 merged 2 commits into
masterfrom
feat/drop-nam-dict-familiar-forms

Conversation

@derek73

@derek73 derek73 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Removes the 736 familiar-form pairs copied from Jörg Michael's nam_dict (GFDL) before any release ships them, keeping the ODU nicknames list's English pairs. This carries out the rule recorded in #1370: a compiler's list copied whole keeps its licence, and the core will not carry the GFDL (decisions/shipped-data-licences, 2026-11-12 #1370 FACTS WE MEASURE).

Rebased onto master after #1370 merged.

What changes

  • tools/vocab/familiar.py writes the nicknames list alone into en.toml. The nam_dict reader, the per-country placement, the bands notes and the de-duplication against nam_dict are removed.
  • 563 pairs leave, the ones the nicknames list doesn't hold, across 20 vocabulary files. mt.toml and sq.toml held nothing else, so they are deleted, and hu.toml now holds exclusions only. FAMILIAR goes from 2,743 pairs of 2,665 names to 2,180 pairs of 2,015 names.
  • familiar-form is English-only until other languages' familiar forms are rebuilt from pairs found in CC0 data and reviewed by hand. That rebuild is a separate follow-up PR.
  • LICENSE-nam_dict's header and README's License bullet now name only _order_names.py's _COMMON_GIVEN block. GFDL-1.2-no-invariants-or-later stays in license, license-files and _THIRD_PARTY. Whichever merges second, this PR or the one replacing _COMMON_GIVEN from ORCID, deletes the file and those three entries.
  • Vocabulary notes that quote a nam_dict frequency band as a bearer count stay. A band is a measured fact, not a copied list.
  • Decisions:
    • name-matching gets a new entry, 2026-11-12 ENGLISH ONLY UNTIL REBUILT. It amends the Standing line and marks the 2026-11-08 and 2026-11-09 entries as superseded in part.
    • shipped-data-licences gets a Log entry recording the drop.
    • group-suggestion's Standing clause is corrected.
  • Docs and the release log: docs/matching.rst, docs/customize.rst, docs/locales.rst, the vocab README and root AGENTS.md now use English examples. The Parser.matching release bullet says the shipped pairs are English only.

Measured

match_diff.py --base #1370's tip (848c00c), 31,212 pairs × 21 settings:

  • The key guarantee holds: no consistent pair is left without a shared key.
  • 4,444 of 655,452 rows move: 44 under each of the 17 settings that don't accept familiar-form, and 919–929 under each of the four that do.
  • 99 rows go from consistent to inconsistent, all under settings accepting familiar-form (Masha/Maria, Józsi/József). 924 given rows read conflict where they read familiar-form. Keys move on 3,532 rows.

No parse moves, since no parse reads a familiar reading.

173 of nam_dict's pairs are also in the nicknames list, and they now ship as the list's readings. The list's review read only the 2,624 rows nam_dict didn't already hold. Both this change's reviewer and I read those 173 by hand and found none in a refused category.

Tests

  • Full suite: 46,796 passed. ruff, mypy, sphinx-build -b doctest and the README doctest are clean.
  • test_familiar_tool.py is rewritten for the list alone. A new test checks that every shipped familiar reading is the list's and sits in en.toml.
  • test_match.py plants its Hungarian accent pairs in a caller's lexicon instead of relying on shipped pairs.
  • A new planted-index test covers _reading_index.tags()'s filter for a file of familiar readings alone, whose last shipped examples were mt.toml and sq.toml. Negative control: it fails with the filter removed.

Reviews run

  • design-docs-reviewer on the docs/design and AGENTS.md changes: five findings, all fixed.
  • pr-review-toolkit code-reviewer on the change: one coverage gap and stale prose, all fixed.
  • code-reviewer on the fix commit: no defects at 80 or above; three minor items applied.

🤖 Generated with Claude Code

@derek73 derek73 added vocabulary Fixed by a word-list entry: per-word vocabulary and language packs matching Name matching: Parser.matching(), check() and keys() labels Nov 13, 2026
@derek73 derek73 self-assigned this Nov 13, 2026
@derek73
derek73 deleted the branch fix/licence-metadata-familiar-data November 13, 2026 05:53
Base automatically changed from fix/licence-metadata-familiar-data to master November 13, 2026 05:53
@derek73 derek73 closed this Nov 13, 2026
derek73 and others added 2 commits November 12, 2026 21:59
A compiler's list copied whole keeps its licence, and nam_dict's is
the GFDL, which the core will not carry (decisions/shipped-data-licences,
2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave
before any release ships them, 563 of them not in the nicknames list;
the familiar-form outcome reads English pairs alone until other
languages' pairs are rebuilt from CC0 data and reviewed by hand.

- tools/vocab/familiar.py writes the nicknames list alone
- the vocabulary and its compiled copies lose nam_dict's readings;
  mt.toml and sq.toml, which held nothing else, go
- LICENSE-nam_dict and README name the suggester's common given
  names alone; the GFDL stays in the metadata until those go too
- tests, docs/matching.rst, customize.rst and the release log move
  to English examples; decisions/name-matching records the measure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- plant a file of familiar readings alone to test _reading_index.tags()'s
  filter, whose last shipped witnesses were mt.toml and sq.toml
- Maximilian / Maxwell for the never-chained example, Max / Maxim no
  longer being a shipped pair
- name every part of the 2026-11-09 entry the drop supersedes, record
  that the 173 shared pairs skipped the list's review, and give the
  match_diff figures per setting as the tool printed them
- docs/locales.rst, group-suggestion's Standing line, the licence
  entry's issue and Standing tense, and stale comments
- AGENTS.md: a filter whose only witnesses are shipped files

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 reopened this Nov 13, 2026
@derek73
derek73 force-pushed the feat/drop-nam-dict-familiar-forms branch from d9971df to 7a5cd42 Compare November 13, 2026 06:04
@derek73 derek73 added this to the 3.1 milestone Nov 13, 2026
@derek73
derek73 merged commit 73718fd into master Nov 13, 2026
14 checks passed
@derek73
derek73 deleted the feat/drop-nam-dict-familiar-forms branch November 13, 2026 06:16
derek73 added a commit that referenced this pull request Nov 13, 2026
…ict (#1374)

* Name the licences of the shipped data in the package metadata

The licence expression read LGPL-2.1-or-later alone while the wheel
ships data under five other licences. It now reads
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later
AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.

- LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which
  its terms ask to travel with the familiar forms and the suggester's
  common given names and which no file in the tree held.
- LICENSE-unicode: the Unicode License v3 notice for the Unihan data,
  in locales/zh.py's simplified surname forms since 2.5.0 and in the
  suggester's pinyin surnames; zh.py's docstring now credits it.
- README.rst's License section lists every source.
- tests/v2/test_licensing.py holds the licence files, license-files,
  the expression, the README and the crediting modules together.
- decisions/shipped-data-licences records the sources, what each
  licence asks and how it is met; AGENTS.md says what a new source
  needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cite #1370 in the licence entry and the release log

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record which shipped data keeps its source's licence

Data the project derives by measuring is its own, its sources credited
as a courtesy; a curated list copied whole keeps its licence. Under
that rule nam_dict's familiar-form pairs are dropped before 3.1 and
rebuilt from CC0 data reviewed by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Replace Unihan with Wikidata for the ZH simplified forms and the pinyin surnames

locales.ZH's simplified surname forms now come from the simplified-
Chinese labels of Wikidata's family-name items, and the group
suggester's pinyin surnames from those items' Hanyu Pinyin statements,
both CC0, so the package ships no Unihan-derived data and Unicode-3.0
and LICENSE-unicode leave its licence metadata.

Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave
locales.ZH; over 322,895 corpus names holding a Han character, 9 move,
all foreign names the pack had cut after 盖. The pinyin block holds
the surname readings (曾 zeng, 单 shan) where Unihan's first reading
gave ceng and dan; suggest_measure moved 37 cells, no quoted figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Key the Wikidata caches by their query, refuse a cut answer, and fix the review's doc findings

Both tools reused a cached answer whenever it covered the forms asked,
whatever the query; each now reuses only the answer to the exact
queries it writes, asks in batches of 100 (the whole set outran the
service's time limit), and refuses an answer holding a row that is not
one, which is how the service's error text reads when it cuts a CSV
answer short. A form an item writes the same in both scripts now
counts as its own simplified form, as Unihan's table listed it, so it
leaves the form two candidates; the shipped blocks do not move.

The group-suggestion entry counted listing lines, not cells (59 moved,
54 up, 5 down), left out `an` and named two anti-goal rows of three;
the #695 and TABLES entries get head notes, AGENTS.md says Unihan has
left, and the release-log bullet says what 2.5.0-3.0.1 declared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Stub the copy of name_usage the pinyin loader calls, and let a pinyin batch be empty

The order_names tests patched tool.name_usage, but ask() calls the
name_usage zh_surnames holds, which another test module's loader can
make a different object, sending the tests to Wikidata; they now patch
that copy and refuse the network. The pinyin cache test now asks a
subset of what it cached, the case the parent answered from the cache,
and a changed pinyin query. A batch of 100 items may hold none with a
reading, so an empty pinyin batch is an answer. W1's Open line records
that #981's mainland ranking would restore the seven dropped forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rank the suggester's common given names from ORCID's given-name boxes, not nam_dict

_COMMON_GIVEN now ranks, per country an ORCID holder states, the words
typed first in the given-name box (CC0), with the same family-share
filter, a minimum of 3 uses in the country, and two refusals the new
source needs: a word holding a full stop, and a word the default
lexicon reads only as a title (Prof., Dra., Lic. typed before the
name). The table holds 6,407 names where nam_dict's held 4,262.

suggest_measure: 158 cells moved, 146 up and 12 down (at most 0.6
points); PatentsView's Indian inventors are read right in 95% of groups
from 50 names where it took 200, and Indian and Nigerian groups gain up
to 25 points at 10 and 20 names. The worst same-order row outside the
anti-goal rows still reads 98%.

LICENSE-nam_dict now names the familiar forms alone; GFDL stays in the
expression for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove LICENSE-unicode again, which merging the squashed #1370 restored

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Key the given names as the suggester looks them up, and fix the review's figures

The tool stripped full stops and commas from a word, where the
suggester strips quotes and brackets first, so `ala'` entered a table
the suggester could never match it in; `spelling()` now keys a word as
the suggester reads it, with a test holding the two together. The
table holds 6,409 names; the floor rises to 5,000, a third of a cut
input. listed_titles' docstring says what it computes, and the filter
docstrings name the title refusal. The given-name Standing line is
proposed, for review; the entry counts 159 cells (147 up) and 12
parliament lists, records the anti-goal rows that rose, `mst`, and the
#1370 entry gets a head note; Parser.suggest quotes the re-taken 30%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hold the spelling test to the suggester's own path, and record mst and ace as an accepted cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Note that the GIVEN NAMES FROM ORCID entry supersedes #1372's licence file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say the familiar pairs were dropped, not rebuilt, in the licence bullet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Show Wang Xiaoming's given-name vote in the suggest doctests

The ORCID table lists `xiaoming`, so `print(suggestion)` in
getting-started.rst and suggest.rst counts two given-name votes for
family first, where the expected output said one; the Sphinx doctest
step failed in every build job. AGENTS.md now says a suggester table
change needs that step run too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

matching Name matching: Parser.matching(), check() and keys() vocabulary Fixed by a word-list entry: per-word vocabulary and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant