Repository navigation
Name the licences of the shipped data in the package metadata - #1370
Merged
Merged
Conversation
The licence expression read LGPL-2.1-or-later alone while the wheel ships data under five other licences. It now reads LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0. - LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which its terms ask to travel with the familiar forms and the suggester's common given names and which no file in the tree held. - LICENSE-unicode: the Unicode License v3 notice for the Unihan data, in locales/zh.py's simplified surname forms since 2.5.0 and in the suggester's pinyin surnames; zh.py's docstring now credits it. - README.rst's License section lists every source. - tests/v2/test_licensing.py holds the licence files, license-files, the expression, the README and the crediting modules together. - decisions/shipped-data-licences records the sources, what each licence asks and how it is met; AGENTS.md says what a new source needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data the project derives by measuring is its own, its sources credited as a courtesy; a curated list copied whole keeps its licence. Under that rule nam_dict's familiar-form pairs are dropped before 3.1 and rebuilt from CC0 data reviewed by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
Merged
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
A compiler's list copied whole keeps its licence, and nam_dict's is the GFDL, which the core will not carry (decisions/shipped-data-licences, 2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave before any release ships them, 563 of them not in the nicknames list; the familiar-form outcome reads English pairs alone until other languages' pairs are rebuilt from CC0 data and reviewed by hand. - tools/vocab/familiar.py writes the nicknames list alone - the vocabulary and its compiled copies lose nam_dict's readings; mt.toml and sq.toml, which held nothing else, go - LICENSE-nam_dict and README name the suggester's common given names alone; the GFDL stays in the metadata until those go too - tests, docs/matching.rst, customize.rst and the release log move to English examples; decisions/name-matching records the measure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
A compiler's list copied whole keeps its licence, and nam_dict's is the GFDL, which the core will not carry (decisions/shipped-data-licences, 2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave before any release ships them, 563 of them not in the nicknames list; the familiar-form outcome reads English pairs alone until other languages' pairs are rebuilt from CC0 data and reviewed by hand. - tools/vocab/familiar.py writes the nicknames list alone - the vocabulary and its compiled copies lose nam_dict's readings; mt.toml and sq.toml, which held nothing else, go - LICENSE-nam_dict and README name the suggester's common given names alone; the GFDL stays in the metadata until those go too - tests, docs/matching.rst, customize.rst and the release log move to English examples; decisions/name-matching records the measure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
* Drop nam_dict's familiar forms, keeping the nicknames list's (#1370) A compiler's list copied whole keeps its licence, and nam_dict's is the GFDL, which the core will not carry (decisions/shipped-data-licences, 2026-11-12 #1370 FACTS WE MEASURE). Its 736 familiar-form pairs leave before any release ships them, 563 of them not in the nicknames list; the familiar-form outcome reads English pairs alone until other languages' pairs are rebuilt from CC0 data and reviewed by hand. - tools/vocab/familiar.py writes the nicknames list alone - the vocabulary and its compiled copies lose nam_dict's readings; mt.toml and sq.toml, which held nothing else, go - LICENSE-nam_dict and README name the suggester's common given names alone; the GFDL stays in the metadata until those go too - tests, docs/matching.rst, customize.rst and the release log move to English examples; decisions/name-matching records the measure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the review's findings on dropping nam_dict's familiar forms - plant a file of familiar readings alone to test _reading_index.tags()'s filter, whose last shipped witnesses were mt.toml and sq.toml - Maximilian / Maxwell for the never-chained example, Max / Maxim no longer being a shipped pair - name every part of the 2026-11-09 entry the drop supersedes, record that the 173 shared pairs skipped the list's review, and give the match_diff figures per setting as the tool printed them - docs/locales.rst, group-suggestion's Standing line, the licence entry's issue and Standing tense, and stale comments - AGENTS.md: a filter whose only witnesses are shipped files Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
…e suggester's pinyin (#1371) * Name the licences of the shipped data in the package metadata The licence expression read LGPL-2.1-or-later alone while the wheel ships data under five other licences. It now reads LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0. - LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which its terms ask to travel with the familiar forms and the suggester's common given names and which no file in the tree held. - LICENSE-unicode: the Unicode License v3 notice for the Unihan data, in locales/zh.py's simplified surname forms since 2.5.0 and in the suggester's pinyin surnames; zh.py's docstring now credits it. - README.rst's License section lists every source. - tests/v2/test_licensing.py holds the licence files, license-files, the expression, the README and the crediting modules together. - decisions/shipped-data-licences records the sources, what each licence asks and how it is met; AGENTS.md says what a new source needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cite #1370 in the licence entry and the release log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record which shipped data keeps its source's licence Data the project derives by measuring is its own, its sources credited as a courtesy; a curated list copied whole keeps its licence. Under that rule nam_dict's familiar-form pairs are dropped before 3.1 and rebuilt from CC0 data reviewed by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Replace Unihan with Wikidata for the ZH simplified forms and the pinyin surnames locales.ZH's simplified surname forms now come from the simplified- Chinese labels of Wikidata's family-name items, and the group suggester's pinyin surnames from those items' Hanyu Pinyin statements, both CC0, so the package ships no Unihan-derived data and Unicode-3.0 and LICENSE-unicode leave its licence metadata. Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave locales.ZH; over 322,895 corpus names holding a Han character, 9 move, all foreign names the pack had cut after 盖. The pinyin block holds the surname readings (曾 zeng, 单 shan) where Unihan's first reading gave ceng and dan; suggest_measure moved 37 cells, no quoted figure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Key the Wikidata caches by their query, refuse a cut answer, and fix the review's doc findings Both tools reused a cached answer whenever it covered the forms asked, whatever the query; each now reuses only the answer to the exact queries it writes, asks in batches of 100 (the whole set outran the service's time limit), and refuses an answer holding a row that is not one, which is how the service's error text reads when it cuts a CSV answer short. A form an item writes the same in both scripts now counts as its own simplified form, as Unihan's table listed it, so it leaves the form two candidates; the shipped blocks do not move. The group-suggestion entry counted listing lines, not cells (59 moved, 54 up, 5 down), left out `an` and named two anti-goal rows of three; the #695 and TABLES entries get head notes, AGENTS.md says Unihan has left, and the release-log bullet says what 2.5.0-3.0.1 declared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stub the copy of name_usage the pinyin loader calls, and let a pinyin batch be empty The order_names tests patched tool.name_usage, but ask() calls the name_usage zh_surnames holds, which another test module's loader can make a different object, sending the tests to Wikidata; they now patch that copy and refuse the network. The pinyin cache test now asks a subset of what it cached, the case the parent answered from the cache, and a changed pinyin query. A batch of 100 items may hold none with a reading, so an empty pinyin batch is an answer. W1's Open line records that #981's mainland ranking would restore the seven dropped forms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove LICENSE-unicode again, which merging the squashed #1370 restored Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Nov 13, 2026
…ict (#1374) * Name the licences of the shipped data in the package metadata The licence expression read LGPL-2.1-or-later alone while the wheel ships data under five other licences. It now reads LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0. - LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which its terms ask to travel with the familiar forms and the suggester's common given names and which no file in the tree held. - LICENSE-unicode: the Unicode License v3 notice for the Unihan data, in locales/zh.py's simplified surname forms since 2.5.0 and in the suggester's pinyin surnames; zh.py's docstring now credits it. - README.rst's License section lists every source. - tests/v2/test_licensing.py holds the licence files, license-files, the expression, the README and the crediting modules together. - decisions/shipped-data-licences records the sources, what each licence asks and how it is met; AGENTS.md says what a new source needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cite #1370 in the licence entry and the release log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record which shipped data keeps its source's licence Data the project derives by measuring is its own, its sources credited as a courtesy; a curated list copied whole keeps its licence. Under that rule nam_dict's familiar-form pairs are dropped before 3.1 and rebuilt from CC0 data reviewed by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Replace Unihan with Wikidata for the ZH simplified forms and the pinyin surnames locales.ZH's simplified surname forms now come from the simplified- Chinese labels of Wikidata's family-name items, and the group suggester's pinyin surnames from those items' Hanyu Pinyin statements, both CC0, so the package ships no Unihan-derived data and Unicode-3.0 and LICENSE-unicode leave its licence metadata. Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave locales.ZH; over 322,895 corpus names holding a Han character, 9 move, all foreign names the pack had cut after 盖. The pinyin block holds the surname readings (曾 zeng, 单 shan) where Unihan's first reading gave ceng and dan; suggest_measure moved 37 cells, no quoted figure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Key the Wikidata caches by their query, refuse a cut answer, and fix the review's doc findings Both tools reused a cached answer whenever it covered the forms asked, whatever the query; each now reuses only the answer to the exact queries it writes, asks in batches of 100 (the whole set outran the service's time limit), and refuses an answer holding a row that is not one, which is how the service's error text reads when it cuts a CSV answer short. A form an item writes the same in both scripts now counts as its own simplified form, as Unihan's table listed it, so it leaves the form two candidates; the shipped blocks do not move. The group-suggestion entry counted listing lines, not cells (59 moved, 54 up, 5 down), left out `an` and named two anti-goal rows of three; the #695 and TABLES entries get head notes, AGENTS.md says Unihan has left, and the release-log bullet says what 2.5.0-3.0.1 declared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Stub the copy of name_usage the pinyin loader calls, and let a pinyin batch be empty The order_names tests patched tool.name_usage, but ask() calls the name_usage zh_surnames holds, which another test module's loader can make a different object, sending the tests to Wikidata; they now patch that copy and refuse the network. The pinyin cache test now asks a subset of what it cached, the case the parent answered from the cache, and a changed pinyin query. A batch of 100 items may hold none with a reading, so an empty pinyin batch is an answer. W1's Open line records that #981's mainland ranking would restore the seven dropped forms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rank the suggester's common given names from ORCID's given-name boxes, not nam_dict _COMMON_GIVEN now ranks, per country an ORCID holder states, the words typed first in the given-name box (CC0), with the same family-share filter, a minimum of 3 uses in the country, and two refusals the new source needs: a word holding a full stop, and a word the default lexicon reads only as a title (Prof., Dra., Lic. typed before the name). The table holds 6,407 names where nam_dict's held 4,262. suggest_measure: 158 cells moved, 146 up and 12 down (at most 0.6 points); PatentsView's Indian inventors are read right in 95% of groups from 50 names where it took 200, and Indian and Nigerian groups gain up to 25 points at 10 and 20 names. The worst same-order row outside the anti-goal rows still reads 98%. LICENSE-nam_dict now names the familiar forms alone; GFDL stays in the expression for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Remove LICENSE-unicode again, which merging the squashed #1370 restored Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Key the given names as the suggester looks them up, and fix the review's figures The tool stripped full stops and commas from a word, where the suggester strips quotes and brackets first, so `ala'` entered a table the suggester could never match it in; `spelling()` now keys a word as the suggester reads it, with a test holding the two together. The table holds 6,409 names; the floor rises to 5,000, a third of a cut input. listed_titles' docstring says what it computes, and the filter docstrings name the title refusal. The given-name Standing line is proposed, for review; the entry counts 159 cells (147 up) and 12 parliament lists, records the anti-goal rows that rose, `mst`, and the #1370 entry gets a head note; Parser.suggest quotes the re-taken 30%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Hold the spelling test to the suggester's own path, and record mst and ace as an accepted cost Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Note that the GIVEN NAMES FROM ORCID entry supersedes #1372's licence file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say the familiar pairs were dropped, not rebuilt, in the licence bullet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Show Wang Xiaoming's given-name vote in the suggest doctests The ORCID table lists `xiaoming`, so `print(suggestion)` in getting-started.rst and suggest.rst counts two given-name votes for family first, where the expected output said one; the Sphinx doctest step failed in every build job. AGENTS.md now says a suggester table change needs that step run too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The package's licence expression read
LGPL-2.1-or-lateralone, while the wheel ships data derived from five third-party datasets under other licences. Two of those licences ask their text or notice to travel with copies, and the tree held neither text.Licence expression:
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.packagingcanonicalizes it unchanged, and the wheel and sdist carry it asLicense-Expression(Metadata-Version 2.4).LICENSE-nam_dict(new): nam_dict's copyright and licence notices as the file states them, then the GFDL 1.2 text. It covers the 3.1 familiar forms and the suggester's common given names.LICENSE-unicode(new): the Unicode License v3 notice. It coverslocales/zh.py's simplified surname forms, which have shipped since 2.5.0: 19 of the block's 324 forms come only from Unihan's variant table. It also covers the 3.1 suggester's pinyin surnames. zh.py's docstring now credits Unihan.README.rst's License section lists every source and where its terms are met. This is the PyPI page.
tests/v2/test_licensing.pyholds the licence files,license-files, the expression, the README and the modules crediting the attribution-only sources (CC BY 4.0, OGDL) to one another. Checked against deliberate breakages: droppingLICENSE-unicodefromlicense-files,CC-BY-4.0from the expression, and zh.py's README credit each fail it.decisions/shipped-data-licences(new key) records:SUFFIX_ACRONYMS' Wikipedia origin;AGENTS.md's Extension Patterns now says what a new data source needs.
The code's licence is unchanged. Releases 2.5.0 through 3.0.1 shipped OGDL and Unihan data, and 3.0.0 onward CC BY data, behind the LGPL-only expression. Debian's
debian/copyrightwill need a paragraph per source when 3.1 is packaged.The entry also records the rule for future sources (FACTS WE MEASURE): data the project derives by measuring (its own counts, cutoffs and filters) is its own, with sources credited as a courtesy. A curated list copied whole, one compiler's judgements of which entries belong, keeps its licence, and if that licence is a burden it goes in an optional extra package instead of the core. Whether the attribution-only terms stay in the expression under that rule is recorded as Open; they stay for now.
Follow-ups, each its own PR:
Each removal shrinks the expression and the files again, and the test flags any of them left behind.
Review
design-docs-reviewer on the decisions entry, the AGENTS.md paragraph and the licence-file headers. It found:
All are fixed in this commit. The missing PR number is added in a follow-up commit.
Test plan
uv run pytest: 46,722 passed, 38 skipped, 3 xfaileduv run ruff check, anduv run mypyon the new testsphinx-build -b doctest docs,python -m doctest README.rst, andreadme_rendereron README.rstuv build: the wheel'sdist-info/licenses/and the sdist carry LICENSE, AUTHORS, LICENSE-nicknames, LICENSE-nam_dict and LICENSE-unicode🤖 Generated with Claude Code