Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2

**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and, for `Lexicon` to read it, `_lexicon._COMPILED_FIELDS` and `_LexiconSets` (no module re-exports a compiled set since 3.0, #1007). A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller states a property on a reading (`add(w, kind="salutation", after_given=True)`), so a caller's word reaches the base and the sub-set together, as a file's does (3.0's 4i-b). Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`, spelled `remove("dean", kind="given")` since 3.0's 4i-b), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read by three rosters: two in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`, over `_LISTED_FIELDS`, the roster less four fields) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`, those less `surnames` and `bound_given_names`), and `_assemble._listed_whole`, which reads it whole to tell a word the parse's own lexicon lists for `Reading.languages`. A new field joins all three by default; a field whose words may be names leaves the two in `_vocab`, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too.

**Shipping data from a new source: name its licence in the package metadata.** Data the project derives by measuring (counts, its own cutoffs and filters) is its own; a curated list copied whole keeps its licence, and goes in an extra package rather than the core if that licence is a burden (decisions/shipped-data-licences, FACTS WE MEASURE). A list derived from a third-party dataset (a census table, a name dictionary, a Unicode database) that lands in `nameparser/` adds that dataset's SPDX identifier to `pyproject.toml`'s `license` expression and a line to README.rst's License section, and, where its licence asks its text or notice to travel with copies, a `LICENSE-<source>` file at the root, listed in `license-files` and in `tests/v2/test_licensing.py`'s `_THIRD_PARTY`; a module docstring crediting the source is not enough on its own. The INE (CC BY 4.0, 3.0.0), Taiwan (OGDL, 2.5.0) and Unihan (Unicode License v3, 2.5.0, in `locales/zh.py` through `tools/zh_surnames.py`, which zh.py's docstring did not name) data shipped behind an expression naming the LGPL alone, and nam_dict's GFDL data reached the tree with no copy of the GFDL, until decisions/shipped-data-licences (2026-11-12, #1370).
**Shipping data from a new source: name its licence in the package metadata.** Data the project derives by measuring (counts, its own cutoffs and filters) is its own; a curated list copied whole keeps its licence, and goes in an extra package rather than the core if that licence is a burden (decisions/shipped-data-licences, FACTS WE MEASURE). A list derived from a third-party dataset (a census table, a name dictionary, a Unicode database) that lands in `nameparser/` adds that dataset's SPDX identifier to `pyproject.toml`'s `license` expression and a line to README.rst's License section, and, where its licence asks its text or notice to travel with copies, a `LICENSE-<source>` file at the root, listed in `license-files` and in `tests/v2/test_licensing.py`'s `_THIRD_PARTY`; a module docstring crediting the source is not enough on its own. The INE (CC BY 4.0, 3.0.0), Taiwan (OGDL, 2.5.0) and Unihan (Unicode License v3, 2.5.0, in `locales/zh.py` through `tools/zh_surnames.py`, which zh.py's docstring did not name) data shipped behind an expression naming the LGPL alone, and nam_dict's GFDL data reached the tree with no copy of the GFDL, until decisions/shipped-data-licences (2026-11-12, #1370); the Unihan data has since been replaced by Wikidata's, CC0, and its licence left the expression (that file's 2026-11-12 UNIHAN REPLACED).

**Comparing configurations: use `tools/differential/config_diff.py`, not a throwaway script.** Parsing the corpora under two or more configurations in one tree (the default against a choice, plain against strict, a lexicon with a word removed via `expr:`) is `config_diff.py diff`, parallel, with corpus, country, regex and sample filters, the moves grouped by field and the words whose role changed ranked; `config_diff.py gaps` is the per-language strict-gap report on top of it, and `--readings` writes every name's readings so two trees' runs compare line by line. #1000, #1005, #1017 and #1019 each wrote a single-process script for this (tools/differential/README.md, "Measuring a change").

Expand Down
55 changes: 0 additions & 55 deletions LICENSE-unicode

This file was deleted.

5 changes: 1 addition & 4 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -305,9 +305,6 @@ under other licences, which apply to those parts:
- the group suggester's common given names from Jörg Michael's
nam_dict: GNU Free Documentation License 1.2 or later, in
``LICENSE-nam_dict``;
- ``locales.ZH``'s simplified surname forms and the group suggester's
pinyin surnames from the Unicode Consortium's Unihan database: Unicode
License v3, in ``LICENSE-unicode``;
- ``locales.ES`` and ``locales.ES_AR``'s given names from Spain's
Instituto Nacional de Estadística: Creative Commons Attribution 4.0,
credited in ``nameparser/locales/_es_given_names.py`` and
Expand All @@ -317,7 +314,7 @@ under other licences, which apply to those parts:
``nameparser/locales/zh.py``.

The package's licence expression is
``LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0``.
``LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later AND CC-BY-4.0 AND OGDL-Taiwan-1.0``.

.. |Build Status| image:: https://github.com/derek73/python-nameparser/actions/workflows/python-package.yml/badge.svg
:target: https://github.com/derek73/python-nameparser/actions/workflows/python-package.yml
Expand Down
Loading
Loading