Skip to content

Name the licences of the shipped data in the package metadata - #1370

Merged
derek73 merged 3 commits into
masterfrom
fix/licence-metadata-familiar-data
Nov 13, 2026
Merged

derek73 merged 3 commits into
masterfrom
fix/licence-metadata-familiar-data

Conversation

@derek73

@derek73 derek73 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

The package's licence expression read LGPL-2.1-or-later alone, while the wheel ships data derived from five third-party datasets under other licences. Two of those licences ask their text or notice to travel with copies, and the tree held neither text.

  • Licence expression: LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0. packaging canonicalizes it unchanged, and the wheel and sdist carry it as License-Expression (Metadata-Version 2.4).

  • LICENSE-nam_dict (new): nam_dict's copyright and licence notices as the file states them, then the GFDL 1.2 text. It covers the 3.1 familiar forms and the suggester's common given names.

  • LICENSE-unicode (new): the Unicode License v3 notice. It covers locales/zh.py's simplified surname forms, which have shipped since 2.5.0: 19 of the block's 324 forms come only from Unihan's variant table. It also covers the 3.1 suggester's pinyin surnames. zh.py's docstring now credits Unihan.

  • README.rst's License section lists every source and where its terms are met. This is the PyPI page.

  • tests/v2/test_licensing.py holds the licence files, license-files, the expression, the README and the modules crediting the attribution-only sources (CC BY 4.0, OGDL) to one another. Checked against deliberate breakages: dropping LICENSE-unicode from license-files, CC-BY-4.0 from the expression, and zh.py's README credit each fail it.

  • decisions/shipped-data-licences (new key) records:

    • each source, its first release, what its licence asks and how that is met;
    • what is left out and why: CC0 sources, corpora used only for counts, and census lists with no stated terms;
    • an open question: SUFFIX_ACRONYMS' Wikipedia origin;
    • relicensing to LGPL-3.0, weighed and not taken.

    AGENTS.md's Extension Patterns now says what a new data source needs.

The code's licence is unchanged. Releases 2.5.0 through 3.0.1 shipped OGDL and Unihan data, and 3.0.0 onward CC BY data, behind the LGPL-only expression. Debian's debian/copyright will need a paragraph per source when 3.1 is packaged.

The entry also records the rule for future sources (FACTS WE MEASURE): data the project derives by measuring (its own counts, cutoffs and filters) is its own, with sources credited as a courtesy. A curated list copied whole, one compiler's judgements of which entries belong, keeps its licence, and if that licence is a burden it goes in an optional extra package instead of the core. Whether the attribution-only terms stay in the expression under that rule is recorded as Open; they stay for now.

Follow-ups, each its own PR:

  • replace Unihan with Wikidata's pinyin;
  • derive the suggester's common given names from ORCID instead of nam_dict;
  • drop nam_dict's familiar-form pairs before 3.1, keeping the ODU list's;
  • rebuild the non-English familiar forms from CC0 data (ORCID, GND), reviewed by hand.

Each removal shrinks the expression and the files again, and the test flags any of them left behind.

Review

  • design-docs-reviewer on the decisions entry, the AGENTS.md paragraph and the licence-file headers. It found:

    • Unihan's 2.5.0 shipping in zh.py;
    • the census lists and the Wikipedia import missing from the entry;
    • a test claim wider than the test;
    • a per-release overstatement;
    • the "rest of nameparser" sentences in the two licence files.

    All are fixed in this commit. The missing PR number is added in a follow-up commit.

Test plan

  • uv run pytest: 46,722 passed, 38 skipped, 3 xfailed
  • uv run ruff check, and uv run mypy on the new test
  • sphinx-build -b doctest docs, python -m doctest README.rst, and readme_renderer on README.rst
  • uv build: the wheel's dist-info/licenses/ and the sdist carry LICENSE, AUTHORS, LICENSE-nicknames, LICENSE-nam_dict and LICENSE-unicode

🤖 Generated with Claude Code

The licence expression read LGPL-2.1-or-later alone while the wheel
ships data under five other licences. It now reads
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later
AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.

- LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which
  its terms ask to travel with the familiar forms and the suggester's
  common given names and which no file in the tree held.
- LICENSE-unicode: the Unicode License v3 notice for the Unihan data,
  in locales/zh.py's simplified surname forms since 2.5.0 and in the
  suggester's pinyin surnames; zh.py's docstring now credits it.
- README.rst's License section lists every source.
- tests/v2/test_licensing.py holds the licence files, license-files,
  the expression, the README and the crediting modules together.
- decisions/shipped-data-licences records the sources, what each
  licence asks and how it is met; AGENTS.md says what a new source
  needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added docs Documentation fixes and updates tests vocabulary Fixed by a word-list entry: per-word vocabulary and language packs labels Nov 13, 2026
@derek73 derek73 self-assigned this Nov 13, 2026
derek73 and others added 2 commits November 12, 2026 20:48
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data the project derives by measuring is its own, its sources credited
as a courtesy; a curated list copied whole keeps its licence. Under
that rule nam_dict's familiar-form pairs are dropped before 3.1 and
rebuilt from CC0 data reviewed by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 3.1 milestone Nov 13, 2026
@derek73
derek73 merged commit d26624e into master Nov 13, 2026
14 checks passed
@derek73
derek73 deleted the fix/licence-metadata-familiar-data branch November 13, 2026 05:53
derek73 added a commit that referenced this pull request Nov 13, 2026
A compiler's list copied whole keeps its licence, and nam_dict's is
the GFDL, which the core will not carry (decisions/shipped-data-licences,
2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave
before any release ships them, 563 of them not in the nicknames list;
the familiar-form outcome reads English pairs alone until other
languages' pairs are rebuilt from CC0 data and reviewed by hand.

- tools/vocab/familiar.py writes the nicknames list alone
- the vocabulary and its compiled copies lose nam_dict's readings;
  mt.toml and sq.toml, which held nothing else, go
- LICENSE-nam_dict and README name the suggester's common given
  names alone; the GFDL stays in the metadata until those go too
- tests, docs/matching.rst, customize.rst and the release log move
  to English examples; decisions/name-matching records the measure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 13, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 13, 2026
A compiler's list copied whole keeps its licence, and nam_dict's is
the GFDL, which the core will not carry (decisions/shipped-data-licences,
2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave
before any release ships them, 563 of them not in the nicknames list;
the familiar-form outcome reads English pairs alone until other
languages' pairs are rebuilt from CC0 data and reviewed by hand.

- tools/vocab/familiar.py writes the nicknames list alone
- the vocabulary and its compiled copies lose nam_dict's readings;
  mt.toml and sq.toml, which held nothing else, go
- LICENSE-nam_dict and README name the suggester's common given
  names alone; the GFDL stays in the metadata until those go too
- tests, docs/matching.rst, customize.rst and the release log move
  to English examples; decisions/name-matching records the measure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 13, 2026
* Drop nam_dict's familiar forms, keeping the nicknames list's (#1370)

A compiler's list copied whole keeps its licence, and nam_dict's is
the GFDL, which the core will not carry (decisions/shipped-data-licences,
2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave
before any release ships them, 563 of them not in the nicknames list;
the familiar-form outcome reads English pairs alone until other
languages' pairs are rebuilt from CC0 data and reviewed by hand.

- tools/vocab/familiar.py writes the nicknames list alone
- the vocabulary and its compiled copies lose nam_dict's readings;
  mt.toml and sq.toml, which held nothing else, go
- LICENSE-nam_dict and README name the suggester's common given
  names alone; the GFDL stays in the metadata until those go too
- tests, docs/matching.rst, customize.rst and the release log move
  to English examples; decisions/name-matching records the measure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the review's findings on dropping nam_dict's familiar forms

- plant a file of familiar readings alone to test _reading_index.tags()'s
  filter, whose last shipped witnesses were mt.toml and sq.toml
- Maximilian / Maxwell for the never-chained example, Max / Maxim no
  longer being a shipped pair
- name every part of the 2026-11-09 entry the drop supersedes, record
  that the 173 shared pairs skipped the list's review, and give the
  match_diff figures per setting as the tool printed them
- docs/locales.rst, group-suggestion's Standing line, the licence
  entry's issue and Standing tense, and stale comments
- AGENTS.md: a filter whose only witnesses are shipped files

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 13, 2026
…e suggester's pinyin (#1371)

* Name the licences of the shipped data in the package metadata

The licence expression read LGPL-2.1-or-later alone while the wheel
ships data under five other licences. It now reads
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later
AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.

- LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which
  its terms ask to travel with the familiar forms and the suggester's
  common given names and which no file in the tree held.
- LICENSE-unicode: the Unicode License v3 notice for the Unihan data,
  in locales/zh.py's simplified surname forms since 2.5.0 and in the
  suggester's pinyin surnames; zh.py's docstring now credits it.
- README.rst's License section lists every source.
- tests/v2/test_licensing.py holds the licence files, license-files,
  the expression, the README and the crediting modules together.
- decisions/shipped-data-licences records the sources, what each
  licence asks and how it is met; AGENTS.md says what a new source
  needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cite #1370 in the licence entry and the release log

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record which shipped data keeps its source's licence

Data the project derives by measuring is its own, its sources credited
as a courtesy; a curated list copied whole keeps its licence. Under
that rule nam_dict's familiar-form pairs are dropped before 3.1 and
rebuilt from CC0 data reviewed by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Replace Unihan with Wikidata for the ZH simplified forms and the pinyin surnames

locales.ZH's simplified surname forms now come from the simplified-
Chinese labels of Wikidata's family-name items, and the group
suggester's pinyin surnames from those items' Hanyu Pinyin statements,
both CC0, so the package ships no Unihan-derived data and Unicode-3.0
and LICENSE-unicode leave its licence metadata.

Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave
locales.ZH; over 322,895 corpus names holding a Han character, 9 move,
all foreign names the pack had cut after 盖. The pinyin block holds
the surname readings (曾 zeng, 单 shan) where Unihan's first reading
gave ceng and dan; suggest_measure moved 37 cells, no quoted figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Key the Wikidata caches by their query, refuse a cut answer, and fix the review's doc findings

Both tools reused a cached answer whenever it covered the forms asked,
whatever the query; each now reuses only the answer to the exact
queries it writes, asks in batches of 100 (the whole set outran the
service's time limit), and refuses an answer holding a row that is not
one, which is how the service's error text reads when it cuts a CSV
answer short. A form an item writes the same in both scripts now
counts as its own simplified form, as Unihan's table listed it, so it
leaves the form two candidates; the shipped blocks do not move.

The group-suggestion entry counted listing lines, not cells (59 moved,
54 up, 5 down), left out `an` and named two anti-goal rows of three;
the #695 and TABLES entries get head notes, AGENTS.md says Unihan has
left, and the release-log bullet says what 2.5.0-3.0.1 declared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Stub the copy of name_usage the pinyin loader calls, and let a pinyin batch be empty

The order_names tests patched tool.name_usage, but ask() calls the
name_usage zh_surnames holds, which another test module's loader can
make a different object, sending the tests to Wikidata; they now patch
that copy and refuse the network. The pinyin cache test now asks a
subset of what it cached, the case the parent answered from the cache,
and a changed pinyin query. A batch of 100 items may hold none with a
reading, so an empty pinyin batch is an answer. W1's Open line records
that #981's mainland ranking would restore the seven dropped forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove LICENSE-unicode again, which merging the squashed #1370 restored

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 13, 2026
…ict (#1374)

* Name the licences of the shipped data in the package metadata

The licence expression read LGPL-2.1-or-later alone while the wheel
ships data under five other licences. It now reads
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later
AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.

- LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which
  its terms ask to travel with the familiar forms and the suggester's
  common given names and which no file in the tree held.
- LICENSE-unicode: the Unicode License v3 notice for the Unihan data,
  in locales/zh.py's simplified surname forms since 2.5.0 and in the
  suggester's pinyin surnames; zh.py's docstring now credits it.
- README.rst's License section lists every source.
- tests/v2/test_licensing.py holds the licence files, license-files,
  the expression, the README and the crediting modules together.
- decisions/shipped-data-licences records the sources, what each
  licence asks and how it is met; AGENTS.md says what a new source
  needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Cite #1370 in the licence entry and the release log

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record which shipped data keeps its source's licence

Data the project derives by measuring is its own, its sources credited
as a courtesy; a curated list copied whole keeps its licence. Under
that rule nam_dict's familiar-form pairs are dropped before 3.1 and
rebuilt from CC0 data reviewed by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Replace Unihan with Wikidata for the ZH simplified forms and the pinyin surnames

locales.ZH's simplified surname forms now come from the simplified-
Chinese labels of Wikidata's family-name items, and the group
suggester's pinyin surnames from those items' Hanyu Pinyin statements,
both CC0, so the package ships no Unihan-derived data and Unicode-3.0
and LICENSE-unicode leave its licence metadata.

Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave
locales.ZH; over 322,895 corpus names holding a Han character, 9 move,
all foreign names the pack had cut after 盖. The pinyin block holds
the surname readings (曾 zeng, 单 shan) where Unihan's first reading
gave ceng and dan; suggest_measure moved 37 cells, no quoted figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Key the Wikidata caches by their query, refuse a cut answer, and fix the review's doc findings

Both tools reused a cached answer whenever it covered the forms asked,
whatever the query; each now reuses only the answer to the exact
queries it writes, asks in batches of 100 (the whole set outran the
service's time limit), and refuses an answer holding a row that is not
one, which is how the service's error text reads when it cuts a CSV
answer short. A form an item writes the same in both scripts now
counts as its own simplified form, as Unihan's table listed it, so it
leaves the form two candidates; the shipped blocks do not move.

The group-suggestion entry counted listing lines, not cells (59 moved,
54 up, 5 down), left out `an` and named two anti-goal rows of three;
the #695 and TABLES entries get head notes, AGENTS.md says Unihan has
left, and the release-log bullet says what 2.5.0-3.0.1 declared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Stub the copy of name_usage the pinyin loader calls, and let a pinyin batch be empty

The order_names tests patched tool.name_usage, but ask() calls the
name_usage zh_surnames holds, which another test module's loader can
make a different object, sending the tests to Wikidata; they now patch
that copy and refuse the network. The pinyin cache test now asks a
subset of what it cached, the case the parent answered from the cache,
and a changed pinyin query. A batch of 100 items may hold none with a
reading, so an empty pinyin batch is an answer. W1's Open line records
that #981's mainland ranking would restore the seven dropped forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rank the suggester's common given names from ORCID's given-name boxes, not nam_dict

_COMMON_GIVEN now ranks, per country an ORCID holder states, the words
typed first in the given-name box (CC0), with the same family-share
filter, a minimum of 3 uses in the country, and two refusals the new
source needs: a word holding a full stop, and a word the default
lexicon reads only as a title (Prof., Dra., Lic. typed before the
name). The table holds 6,407 names where nam_dict's held 4,262.

suggest_measure: 158 cells moved, 146 up and 12 down (at most 0.6
points); PatentsView's Indian inventors are read right in 95% of groups
from 50 names where it took 200, and Indian and Nigerian groups gain up
to 25 points at 10 and 20 names. The worst same-order row outside the
anti-goal rows still reads 98%.

LICENSE-nam_dict now names the familiar forms alone; GFDL stays in the
expression for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Remove LICENSE-unicode again, which merging the squashed #1370 restored

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Key the given names as the suggester looks them up, and fix the review's figures

The tool stripped full stops and commas from a word, where the
suggester strips quotes and brackets first, so `ala'` entered a table
the suggester could never match it in; `spelling()` now keys a word as
the suggester reads it, with a test holding the two together. The
table holds 6,409 names; the floor rises to 5,000, a third of a cut
input. listed_titles' docstring says what it computes, and the filter
docstrings name the title refusal. The given-name Standing line is
proposed, for review; the entry counts 159 cells (147 up) and 12
parliament lists, records the anti-goal rows that rose, `mst`, and the
#1370 entry gets a head note; Parser.suggest quotes the re-taken 30%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Hold the spelling test to the suggester's own path, and record mst and ace as an accepted cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Note that the GIVEN NAMES FROM ORCID entry supersedes #1372's licence file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Say the familiar pairs were dropped, not rebuilt, in the licence bullet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Show Wang Xiaoming's given-name vote in the suggest doctests

The ORCID table lists `xiaoming`, so `print(suggestion)` in
getting-started.rst and suggest.rst counts two given-name votes for
family first, where the expected output said one; the Sphinx doctest
step failed in every build job. AGENTS.md now says a suggester table
change needs that step run too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation fixes and updates tests vocabulary Fixed by a word-list entry: per-word vocabulary and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant