Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
5fbcd20
Name the licences of the shipped data in the package metadata
derek73 Nov 13, 2026
e9cb352
Cite #1370 in the licence entry and the release log
derek73 Nov 13, 2026
848c00c
Record which shipped data keeps its source's licence
derek73 Nov 13, 2026
3bb3f9e
Replace Unihan with Wikidata for the ZH simplified forms and the piny…
derek73 Nov 13, 2026
03292d6
Key the Wikidata caches by their query, refuse a cut answer, and fix …
derek73 Nov 13, 2026
5a71372
Stub the copy of name_usage the pinyin loader calls, and let a pinyin…
derek73 Nov 13, 2026
ff2022a
Rank the suggester's common given names from ORCID's given-name boxes…
derek73 Nov 13, 2026
b101628
Merge remote-tracking branch 'origin/master' into fix/replace-unihan-…
derek73 Nov 13, 2026
bcf2fb2
Remove LICENSE-unicode again, which merging the squashed #1370 restored
derek73 Nov 13, 2026
773b128
Key the given names as the suggester looks them up, and fix the revie…
derek73 Nov 13, 2026
d4d8b99
Merge remote-tracking branch 'origin/master' into fix/replace-unihan-…
derek73 Nov 13, 2026
439386d
Hold the spelling test to the suggester's own path, and record mst an…
derek73 Nov 13, 2026
b6816da
Merge remote-tracking branch 'origin/fix/replace-unihan-with-wikidata…
derek73 Nov 13, 2026
2854018
Note that the GIVEN NAMES FROM ORCID entry supersedes #1372's licence…
derek73 Nov 13, 2026
e2da8ca
Say the familiar pairs were dropped, not rebuilt, in the licence bullet
derek73 Nov 13, 2026
b0f7396
Merge remote-tracking branch 'origin/master' into feat/orcid-common-g…
derek73 Nov 13, 2026
0d98214
Show Wang Xiaoming's given-name vote in the suggest doctests
derek73 Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md

Large diffs are not rendered by default.

426 changes: 0 additions & 426 deletions LICENSE-nam_dict

This file was deleted.

5 changes: 1 addition & 4 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -302,9 +302,6 @@ under other licences, which apply to those parts:

- the familiar forms from Old Dominion University's nicknames list:
Apache License 2.0, in ``LICENSE-nicknames``;
- the group suggester's common given names from Jörg Michael's
nam_dict: GNU Free Documentation License 1.2 or later, in
``LICENSE-nam_dict``;
- ``locales.ES`` and ``locales.ES_AR``'s given names from Spain's
Instituto Nacional de Estadística: Creative Commons Attribution 4.0,
credited in ``nameparser/locales/_es_given_names.py`` and
Expand All @@ -314,7 +311,7 @@ under other licences, which apply to those parts:
``nameparser/locales/zh.py``.

The package's licence expression is
``LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0``.
``LGPL-2.1-or-later AND Apache-2.0 AND CC-BY-4.0 AND OGDL-Taiwan-1.0``.

.. |Build Status| image:: https://github.com/derek73/python-nameparser/actions/workflows/python-package.yml/badge.svg
:target: https://github.com/derek73/python-nameparser/actions/workflows/python-package.yml
Expand Down
11 changes: 8 additions & 3 deletions docs/design/decisions/group-suggestion.md

Large diffs are not rendered by default.

6 changes: 4 additions & 2 deletions docs/design/decisions/shipped-data-licences.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@
- A dataset under CC0, a corpus the package measured counts on without copying its words, and a source that states no terms for reuse add nothing to the expression; the last kind is listed in this file's Log as left out, by name, so the next source of its kind is weighed against it — 2026-11-12 #1370 THE LICENCE EXPRESSION.
- Data the project derives by measuring (counts, rankings, cutoffs and filters of its own over its sources) is the project's own, its sources credited as a courtesy; a curated list copied as a whole, one compiler's judgements of which entries belong, keeps its licence, and a list the core will not carry under that licence ships as an optional extra package, as the Japanese segmenter does, or not at all — 2026-11-12 #1370 FACTS WE MEASURE.
- The familiar forms copied from nam_dict are dropped before 3.1, keeping the ODU list's (done, 2026-11-12 #1370 NAM_DICT'S FAMILIAR PAIRS LEAVE THE PACKAGE), and rebuilt from pairs the project finds in CC0 data and reviews by hand (not yet done) — 2026-11-12 #1370 FACTS WE MEASURE.
- No shipped data derives from nam_dict since 2026-11-12, so the GNU FDL and `LICENSE-nam_dict` left the package before any release named them: its familiar pairs were dropped (#1370 NAM_DICT'S FAMILIAR PAIRS LEAVE THE PACKAGE) and the suggester's common given names are ranked from ORCID (CC0) — 2026-11-12 GIVEN NAMES FROM ORCID.
- No shipped data derives from Unihan since 2026-11-12: `locales/zh.py`'s simplified forms come from the simplified-Chinese labels of Wikidata's family-name items and `_order_names.py`'s pinyin surnames from those items' Hanyu Pinyin statements, both CC0, so `Unicode-3.0` and `LICENSE-unicode` left the package; 2.5.0 through 3.0.1 shipped Unihan-derived forms under the expression naming the LGPL alone — 2026-11-12 UNIHAN REPLACED.

## Log

- **2026-11-12 #1370 NAM_DICT'S FAMILIAR PAIRS LEAVE THE PACKAGE; `LICENSE-nam_dict` NAMES THE GROUP SUGGESTER'S COMMON GIVEN NAMES ALONE.** The first half of the FACTS WE MEASURE ruling below, carried out: the 736 familiar-form pairs `tools/vocab/familiar.py` wrote from nam_dict into 20 vocabulary files are removed, with the generator's nam_dict path, before any release ships them (decisions/name-matching, 2026-11-12 ENGLISH ONLY UNTIL REBUILT). `mt.toml` and `sq.toml`, which held nothing else, go with them. `LICENSE-nam_dict`'s header and README's License bullet now name `config/_order_names.py`'s `_COMMON_GIVEN` block alone, the one part of the package still derived from nam_dict until the suggester's common given names are drawn from ORCID; `GFDL-1.2-no-invariants-or-later` stays in `license`, `license-files` and `tests/v2/test_licensing.py`'s `_THIRD_PARTY` until then, and the change that removes the last of nam_dict's data deletes the file and the three entries. The counts in vocabulary notes that quote a nam_dict frequency band (`nam_dict band 5 in Sweden`) stay: a band is a fact the project measured, not a list it copied.
- **2026-11-12 #1370 THE LICENCE EXPRESSION NAMES EVERY DATASET THE WHEEL SHIPS, AND NAM_DICT AND UNIHAN GET THE LICENCE FILES THEIR TERMS ASK FOR.** Until this entry `license` read `LGPL-2.1-or-later` alone and `license-files` held `LICENSE`, `AUTHORS` and `LICENSE-nicknames` (decisions/name-matching's 2026-11-09 THE NICKNAMES LIST). Five third-party datasets under stated terms reach the package, read from each module's provenance docstring and the tools that write the generated blocks, on master at bda4a632:
- **2026-11-12 #1370 NAM_DICT'S FAMILIAR PAIRS LEAVE THE PACKAGE; `LICENSE-nam_dict` NAMES THE GROUP SUGGESTER'S COMMON GIVEN NAMES ALONE.** (The licence file it leaves superseded by 2026-11-12 GIVEN NAMES FROM ORCID below, which ranks those names from ORCID and deletes `LICENSE-nam_dict` and the GFDL term.) The first half of the FACTS WE MEASURE ruling below, carried out: the 736 familiar-form pairs `tools/vocab/familiar.py` wrote from nam_dict into 20 vocabulary files are removed, with the generator's nam_dict path, before any release ships them (decisions/name-matching, 2026-11-12 ENGLISH ONLY UNTIL REBUILT). `mt.toml` and `sq.toml`, which held nothing else, go with them. `LICENSE-nam_dict`'s header and README's License bullet now name `config/_order_names.py`'s `_COMMON_GIVEN` block alone, the one part of the package still derived from nam_dict until the suggester's common given names are drawn from ORCID; `GFDL-1.2-no-invariants-or-later` stays in `license`, `license-files` and `tests/v2/test_licensing.py`'s `_THIRD_PARTY` until then, and the change that removes the last of nam_dict's data deletes the file and the three entries. The counts in vocabulary notes that quote a nam_dict frequency band (`nam_dict band 5 in Sweden`) stay: a band is a fact the project measured, not a list it copied.
- **2026-11-12 #1370 THE LICENCE EXPRESSION NAMES EVERY DATASET THE WHEEL SHIPS, AND NAM_DICT AND UNIHAN GET THE LICENCE FILES THEIR TERMS ASK FOR.** (The suggester's common given names in its nam_dict row, and ORCID's part in them, superseded by 2026-11-12 GIVEN NAMES FROM ORCID below: ranked from ORCID's given-name boxes, CC0; no release shipped the nam_dict-derived table.) Until this entry `license` read `LGPL-2.1-or-later` alone and `license-files` held `LICENSE`, `AUTHORS` and `LICENSE-nicknames` (decisions/name-matching's 2026-11-09 THE NICKNAMES LIST). Five third-party datasets under stated terms reach the package, read from each module's provenance docstring and the tools that write the generated blocks, on master at bda4a632:

| Source | Shipped in | First release | Licence (SPDX) | What it asks | How it is met |
|---|---|---|---|---|---|
Expand All @@ -34,3 +35,4 @@
- **2026-11-12 UNIHAN REPLACED — THE SIMPLIFIED SURNAME FORMS AND THE PINYIN SURNAMES COME FROM WIKIDATA, AND THE UNICODE LICENCE LEAVES THE PACKAGE.** Of the two shipped derivatives the 2026-11-12 #1370 entry lists from Unihan, neither ships now; that entry's table stays the record of what 2.5.0 through 3.0.1 shipped. This settles FACTS WE MEASURE's Open item for `Unicode-3.0`: with no Unihan-derived data in the package, the term leaves the expression whatever that item decides for derived facts.
- `locales/zh.py`'s simplified forms. `tools/zh_surnames.py` gives a ranked form the simplified form that the simplified-Chinese labels (`zh-hans`, `zh-cn`, `zh-sg`, `zh-my`) of the Wikidata items labelled with it give, where they give exactly one in the Basic Multilingual Plane, the rule Unihan's variant table was read by; the items are every family name (Q101352) and Chinese family name (Q1093580), at any rank, since the items for 關, 顏 and 遲 are family names but not Chinese family names. Of the 19 forms Unihan alone had added (the #1370 entry's list), 12 are attested so (东云兰关区娄盘练迟阙颜风) and 7 are not (买厐盖篮芈诸谌), whose traditional forms no family-name item labels (買, 龎, 籃, 諶) or whose items' simplified labels give no other form (蓋, 諸, 羋); the block holds 317 forms where it held 324. Measured over every name holding a Han character in the committed corpora and the local corpora, read under `plain:ZH` in this tree and in the tree before it (`tools/differential/config_diff.py diff --committed --local all --match '[\u4e00-\u9fff\u3400-\u4dbf]' --readings <file> default plain:ZH` in each, the two files compared line by line): 9 of 322,895 names moved, all transliterations of foreign names opening with 盖 (`盖尔哈尔德`, `盖达尔 阿利耶夫`), which the pack had cut after 盖 and which now read as the default reads them; `cldr_agreement.py`'s records, both locale views included, did not move. A Chinese name with one of the 7 surnames in simplified characters (`谌龙`, `盖丽丽`) is no longer divided; written in traditional characters (`蓋麗麗`) it still is. Weighed and not taken: crediting the form most of an item's bearers write their own simplified labels with (P734), which attests 盖 and 诸 but reads `马` for a European surname item labelled 蓋 and attests none of the other five.
- `config/_order_names.py`'s pinyin surnames. `tools/vocab/order_names.py` takes every Hanyu Pinyin reading that the Wikidata items labelled with a `locales.ZH` single-character surname state, by three statements (P2440 qualified by P459 Hanyu Pinyin, Q42222; the P1721 qualifier of P1705; the P1705 of the item P460 names under the criterion P1013 Q42222), and leaves out a surname no item gives one for, naming each in its run: 35 of 463 on 2026-11-12 (偕储儲兵农刁力塗奚安宫宮寇浦留瞿祝籃繆缪芦蘆褚諶諸買辜農鄞闕阙髙龎𡍼𨶒). The block holds 200 spellings where it held 206: `bu`, `nian`, `shan`, `zeng`, `zha` and `zhai` joined, the surname readings of 卜, 粘, 单, 曾, 查 and 翟, which Unihan's first `kMandarin` reading gave as `bo`, `zhan`, `dan`, `ceng`, `cha` and `di`; `an`, `bing`, `bo`, `ceng`, `cha`, `chou`, `dan`, `diao`, `jie`, `kou`, `nong` and `que` left, the readings of the surnames left out and the readings that were not the surname's (`an` stays in the table through the Korean list). The suggester's figures moved as decisions/group-suggestion's 2026-11-12 PINYIN FROM WIKIDATA entry records.
- **2026-11-12 GIVEN NAMES FROM ORCID — THE SUGGESTER'S COMMON GIVEN NAMES NO LONGER DERIVE FROM NAM_DICT.** `config/_order_names.py`'s `_COMMON_GIVEN` ranks ORCID's given-name boxes by the country each holder states (CC0) where it ranked nam_dict's country columns by frequency band; the filter over ORCID's two boxes is unchanged. The ranking, cutoff and filter are the project's measurement, the first kind FACTS WE MEASURE names, so the table carries no licence of its own. The table and its moves are decisions/group-suggestion's 2026-11-12 GIVEN NAMES FROM ORCID. No release shipped the nam_dict-derived table. With #1372's familiar pairs gone before it, no nam_dict data is left in the package, so this change deletes `LICENSE-nam_dict`, README's bullet, `GFDL-1.2-no-invariants-or-later` in `license` and the file in `license-files`, and `tests/v2/test_licensing.py`'s `_THIRD_PARTY` row; the expression is `LGPL-2.1-or-later AND Apache-2.0 AND CC-BY-4.0 AND OGDL-Taiwan-1.0`, and no release named the GFDL.
5 changes: 3 additions & 2 deletions docs/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ it, and ``suggestion.base`` is the parser you started from:
Suggested for 5 names (base: Parser(Lexicon(default), Policy())):
name_order FAMILY_FIRST 4 for, 0 against, 1 no evidence
surname FAMILY_FIRST 3
given-name FAMILY_FIRST 1
given-name FAMILY_FIRST 2
patronymic_rules unchanged (none) no evidence
initials_as_family unchanged (False) no evidence
capitals_as_family unchanged (False) 5 for, 0 against, 0 no evidence
Expand All @@ -98,7 +98,8 @@ so they are evidence for leaving ``capitals_as_family`` off. Four names
vote for family first: ``Kim``, ``Wang`` and ``Nguyen`` by
the romanized surname in front, ``Garcia Emily`` by the common given
name at the end (``Garcia`` is not one of the surnames ``suggest``
reads). ``Taylor Morgan`` gives no evidence, so the suggested
reads). ``Wang Xiaoming`` carries both kinds, so the surname and
given-name rows count it once each. ``Taylor Morgan`` gives no evidence, so the suggested
parser reads it family first only because the rest of the list is
written that way; nothing in the name itself can confirm that reading.
A name written the other way round that carries evidence (``Emily
Expand Down
2 changes: 1 addition & 1 deletion docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Release Log

nameparser 3.1 is under development. It refuses a ``Lexicon``, ``Parser`` or locale pack pickled by 2.x, which 3.0 loads with a ``DeprecationWarning``.

- **Change the package's licence metadata to name the licences of the data it ships.** The ``License-Expression`` is ``LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0``, where every release through 3.0.1 gave ``LGPL-2.1-or-later`` alone; the code stays LGPL-2.1-or-later, and ``LICENSE-nam_dict`` joins ``LICENSE-nicknames`` in the wheel. README's License section names each source. :issue:`1370`
- **Change the package's licence metadata to name the licences of the data it ships.** The ``License-Expression`` is ``LGPL-2.1-or-later AND Apache-2.0 AND CC-BY-4.0 AND OGDL-Taiwan-1.0``, where every release through 3.0.1 gave ``LGPL-2.1-or-later`` alone; the code stays LGPL-2.1-or-later. No release shipped data derived from nam_dict (GNU FDL), which the 3.1 cycle's familiar forms and suggester table held until the familiar pairs were dropped and the table was ranked from ORCID. README's License section names each source. :issue:`1370`
- **Change locales.ZH's simplified surnames to the ones Wikidata's family-name items write, no longer Unihan's.** ``parser_for(locales.ZH).parse("谌龙")`` gives family ``谌龙``, undivided, where 2.5.0 through 3.0.1 gave family ``谌``, given ``龙``; seven simplified forms no Wikidata item attests (买, 厐, 盖, 篮, 芈, 诸, 谌) left the list, and their traditional forms (``諶龍``) still divide. 2.5.0 through 3.0.1 shipped those forms from Unihan, whose Unicode License v3 their metadata did not name; 3.1 ships no Unihan-derived data. :issue:`1370`
- **Remove loading a Lexicon, Parser or locale pack pickled by 2.x, which 3.0 deprecated.** ``pickle.loads`` of a ``Parser`` pickled by 2.8.0 raises ``ValueError`` saying to re-pickle it under 3.0, where 3.0.x loaded it with a ``DeprecationWarning``; the error for a 2.x ``Lexicon`` spelling now names :doc:`migrate`'s table rather than each word's calls. :issue:`1003`
- **Change PolicyPatch.script_orders to merge by script.** ``Policy().patched(PolicyPatch(script_orders={Script.HAN: GIVEN_FIRST}))`` keeps the Hangul and hiragana entries beside its Han one, where 2.1.0 through 3.0.1 replaced the whole table with the patch's; an empty patch no longer clears the table, which ``Policy(script_orders=())`` on the base parser does. A pack can now add one script's order without erasing a caller's own, and ``parser_for`` warns where two packs set one script differently. :issue:`1237`
Expand Down
10 changes: 6 additions & 4 deletions docs/suggest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ votes do not decide keeps the base parser's value.

Four names vote for family first: ``Kim``, ``Wang`` and ``Nguyen`` by
the romanized surname in front, ``Garcia Emily`` by the common given
name at the end. ``Taylor Morgan`` gives no evidence: it is read family
name at the end; ``Wang Xiaoming`` carries both, so each signal's row
counts it. ``Taylor Morgan`` gives no evidence: it is read family
first because the list is, and nothing in the name can confirm that.

.. _suggest-disagreeing:
Expand All @@ -77,7 +78,7 @@ written, so it takes ``Clark`` for the given name:
Suggested for 6 names (base: Parser(Lexicon(default), Policy())):
name_order FAMILY_FIRST 4 for, 1 against, 1 no evidence
surname FAMILY_FIRST 3
given-name FAMILY_FIRST 1
given-name FAMILY_FIRST 2
given-name GIVEN_FIRST 1
patronymic_rules unchanged (none) no evidence
initials_as_family unchanged (False) no evidence
Expand Down Expand Up @@ -179,10 +180,11 @@ a name that writes no capitals disagrees with a decided switch:

Where it stops. A group is read right only once it carries enough
names with evidence. Measured on groups drawn from real names written
in a known order (2026-11-10, re-taken 2026-11-11), a list written in the base's own order keeps it
in a known order (2026-11-10, re-taken 2026-11-11 and 2026-11-12), a list written in the base's own order keeps it
in at least 98% of groups from 2 to 200 names, but moving the base
takes more names, from 5 for Chinese, Taiwanese, Korean, Czech and
Polish names to 200 for Indian ones. Two lists keep their order less
Polish names to 50 or 200 for Indian ones (US patent inventors and UK
company filers). Two lists keep their order less
often, the capitals in them read as a family name: bylines written with
the initials first (``GH Yoon``), so a group of ten or more is usually
suggested family-first, and names written with the given name in
Expand Down
13 changes: 8 additions & 5 deletions nameparser/_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -362,27 +362,30 @@ def suggest(self, names: Iterable[str]) -> Suggestion:
an empty group, or one with no votes, suggests no change. A
bare str, bytes, a mapping or a non-str item raises TypeError.

Measured 2026-11-10, re-taken 2026-11-11, with
Measured 2026-11-10, re-taken 2026-11-11 and 2026-11-12, with
tools/differential/suggest_measure.py
on groups drawn from real names written in a known order, 300
groups a size (decisions/group-suggestion's 2026-11-11 TRAILING
INITIAL entry holds the table): a group written in this parser's own order kept
INITIAL entry holds the table, its 2026-11-12 GIVEN NAMES FROM
ORCID entry the cells that moved since): a group written in this parser's own order kept
it in at least 98% of groups at every size from 2 to 200 names,
under either base order, save two populations whose capitals the
capitals signal reads as a family name: ORCID's credit
names written as a byline with the initials first (``GH Yoon``)
kept their given-first order in 96% of groups of two and 0% of
groups of fifty, before this gate as after it, and ORCID holders
who typed their given name in capitals (``ANNA Smith``) kept it
in 27% of groups of two and none of five (decisions/
group-suggestion's 2026-11-12 CAPITALS entry). Moving the base takes more names, and
in 30% of groups of two and none of five (decisions/
group-suggestion's 2026-11-12 CAPITALS entry, re-taken in its
GIVEN NAMES FROM ORCID entry). Moving the base takes more names, and
how many depends on what the names carry: groups of Chinese,
Taiwanese, Korean, Czech or Polish names (the romanized surnames
and the common given names) were read right in 95% of groups
from 5 names; German, Dutch, Italian, Spanish, British and
Vietnamese from 10; French, US, Japanese and Turkish from 20;
Nigerian, Romanian, Russian and Brazilian from 50; and Indian
only at 200. A group whose votes fall short of the thresholds keeps
from 50 for US patent inventors, only at 200 for UK company
filers. A group whose votes fall short of the thresholds keeps
the base rather than guess, so a smaller group moves it in fewer
of the draws. ``capitals_as_family``, measured 2026-11-12
(decisions/group-suggestion's CAPITALS entry): ORCID holders who
Expand Down
Loading
Loading