Skip to content

Commit a5a1ce4

Browse files
derek73claude
andcommitted
Deprecate the 2.x Lexicon translation: a 2.x pickle load warns (#754)
The translation of 2.x set spellings and pickles into readings is migration scaffolding: it stays in 3.0 and is deleted in 3.1. A Lexicon pickled by 2.x (and a Parser or locale pack holding one) now loads with a DeprecationWarning, once per Lexicon, attributed to the caller's pickle.loads line, naming the re-pickle remedy. The old spellings already raise, so their per-word fix is unchanged. Records the lifetime and what 3.1 does instead in decisions/vocabulary-model; notes it in docs/migrate.rst, the release log and AGENTS.md's Pickling bullet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent fc8fda1 commit a5a1ce4

6 files changed

Lines changed: 93 additions & 7 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2
250250
- **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in the 2.x `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`, deleted with the facade in 3.0), which cannot be wrong the way a sentence can.
251251
- **Typing/docs**: `from __future__ import annotations`; `frozen=True, slots=True` on every public dataclass; strict-profile mypy flags via per-module overrides in pyproject (`strict = true` itself is not valid per-module). Docstrings state contracts in prose with **no doctest blocks** — `--doctest-modules` makes every example a test; behavior examples go to unit tests per the lean-docs rule. **Document the positive direction of a partial property**: "a non-empty `ambiguities` is a signal to act on" is checkable, while "an empty one means no fork occurred" is a universal negative needing exhaustive verification -- that claim was written twice and falsified twice, at sites the author had not audited.
252252
- **The segmenter contract**: the optional `Parser(segmenter=...)` hook is parse-totality's ONE exception (locales spec section 4). Everything inside that exception is a bug in USER CODE, never a fact about the name, so it is surfaced rather than absorbed: the segmenter's own exceptions propagate, and the two protocol violations the stage can detect for itself — an answer of the wrong type, and one cutting at or past the end of the token it was handed — raise `TypeError`/`ValueError` from `_script_segment` for the same reason. The line to hold when adding a check there: a protocol violation by the segmenter's AUTHOR raises, while an adapter's defense against its own third-party library (`locales/ja.py`'s repertoire, length, reconstruction and score guards) declines with `None`, because what those catch is a fact about the content.
253-
- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: "<release>"}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name).
253+
- **Pickling**: v2 types must round-trip (`Parser` is picklable by construction, and it holds a `Lexicon`; the one qualifier is that a `Parser` pickles iff its segmenter does — see the segmenter bullet above). Every frozen type assigns `_guarded_getstate`/`_guarded_setstate` (`_types.py`) in its class body (`@dataclass(slots=True)` would override inherited pickle methods) — unpickling fails at the LOAD site on field-layout skew, with one widening: a field added after its type shipped carries `metadata={_types.ADDED_IN: "<release>"}` and a default, and an older pickle missing it loads with the default (`Ambiguity.alternative`, 2.7; `Token.reading`, `Reading.languages`, `Reading.second_surname` and `ParsedName.order`, 3.0.0) -- a missing field without the marker, default or not, an unknown field and a renamed one are still refused, each a row in `tests/v2/test_types.py::_REFUSED_SKEWS`, so a new field on a pickled type takes the marker or breaks every older pickle; the shared guard deliberately does NOT re-validate values (pickle is not a security boundary; canonical state comes from a validated instance). `Lexicon` and `LexiconReading` keep their own copy of the guard (layering); a new unpicklable slot type needs the same treatment plus a round-trip test. Since 3.0's 4i-b a `Lexicon` pickles its lists, its masks, its edit log, its choice of languages (`_languages` and `_strict`, 4i-c, marked `ADDED_IN` so a 4i-b pickle loads choosing nothing, a 2.x state translating to no choice) and its base as a SNAPSHOT of the reading index's rows, so a pickle parses as it did where it was written, a snapshot equal to the running release's index loading as the by-reference default again (decisions/vocabulary-model, 2026-11-02 #754, item 5); its private `_sets`, `_cap_map`, `_chosen` and `_files` fields are built at construction and never pickled. `Lexicon`'s copy of the guard admits the `ADDED_IN` marker, read under `_lexicon._ADDED_IN`, the same string, since layering keeps `_lexicon` from importing `_types` (`test_lexicon_marks_an_added_field_with_the_shared_key` holds the two equal), so a new `Lexicon` field takes the marker and a default or breaks older pickles. A 2.x state holds the compiled sets instead, and `_lexicon._translated_2x` translates it onto the nearer named base and a per-word log, loading where the compile gives the sets back and refusing otherwise, naming the fields, so the 2.8 `Lexicon`, `Parser` and locale-pack pickles in `tests/v2/data/lexicon_v28.pickle` load as the 3.0 calls that build them, each `Lexicon` loaded warning with a `DeprecationWarning`: the translation is migration scaffolding, deleted in 3.1 (decisions/vocabulary-model, 2026-11-02 #754, the 2.x translation's lifetime), so build nothing new on it. And unlike the shared guard it DOES re-validate: a load goes through the constructor, which checks the base, refuses an edit not in canonical form, and refuses a list or mask entry that is not already normalized, because an old blob would otherwise load a different lexicon in silence (`particles_ambiguous` changed meaning between v1 and v2 without changing its name).
254254
- **One sanctioned global**: the (future) cached default `Parser`. Lazily cached FROZEN singletons (`Lexicon.default()`'s `functools.cache`, the future default parser) are constants, not state; any second piece of module-level MUTABLE state requires amending the conventions doc, on purpose, in review.
255255
- **Tests**: all v2 tests live in the `tests/v2/` package, one test module per source module plus the cross-cutting ones (`test_reprs.py`, `test_layering.py`, `test_contracts.py`, `test_properties.py`, `test_benchmark.py`, the `cases.py`/`test_cases.py` table, `test_regex_sync.py`, which pins the pipeline's hand-copied patterns against their sources, and `test_snapshot.py`, which holds every corpus name's reading to `tools/differential/snapshot.jsonl` — sharing `_differential_fixtures.py` with `test_differential.py` and `test_corpora.py` for the handles on `tools/differential/`), names stating behavior; the 1.x suite's parsing tests stand ported to `parse()` in `tests/*.py`. Never assert `Lexicon.default()` contents; the narrow sourcing spot-checks in `test_default_sources_v1_vocabulary` that pin the v1→v2 migration contract (e.g. the flipped `particles_ambiguous` model) are the sanctioned exception.
256256

‎docs/design/decisions/vocabulary-model.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -518,3 +518,8 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names)
518518
- **What.** `Lexicon` gains a private slot `_files`, built at construction beside `_sets` and never compared, hashed or pickled: a `_lexicon._ReadingFiles`, one per base, log and choice (`_reading_files`, an `lru_cache`, so equal lexicons share it), hashed by identity. Assemble's two memos, `_languages` and `_part_languages`, take it as a fourth argument, so a hit stays C-level and costs no frame. Its `languages(text, kinds)` answers the default (base `default`, no log, no choice) from the index's rows through `languages_of`, unchanged; a choice with no log from the word's own rows under `choose` (and the strict filter), which is exact since a choice's preference is per word; and a log or a snapshot base from the pool `_built` compiled at construction. Reading the whole pool on every miss was the prototype's first form and cost about 2 ms on the first listed-word parse (`parser_for(locales.PT)`, 0.23 to 2.5 ms), which the per-word path removes. `ParsedName`, `Token` and `Reading` store nothing new, so no `ADDED_IN` marker: a pickled parse keeps the tuple it was parsed with, and an old `Lexicon` or `Parser` pickle builds the slot on load.
519519
- **What moved, measured over the committed corpora's 6,531 names** (on master `854087be` against this branch, Python 3.11.17), only `languages`: under `pt` 309 names, each narrowing (`de` from `es fr it nl pt` to `pt` in 273, `da`, the one-case `E`, `sr`, `dra`, `santa`); under `cs` 11 (`Mgr.` from `en fr` to `cs`; `st.`, `ml.` and `pan`, read only under the choice, from `()` to `cs`); under `es` 299; under strict `en` 109 (`Ma.` from `en fil` to `en`, `Baron` from `de en fr nl` to `en`, `jr.iii` from `en mul` to `en`); under strict `pt` 337, 21 of them the `Ph. D.` pair going to `()`; with no choice and no log none. A check over nine configurations (the default, `pt`, `cs`, `es`, strict `en`, strict `pt`, the default with an `add` and a `remove`, `pt` over a caller's `add`, and `Lexicon.empty()`) found the per-word paths agreeing with the whole pool's answer on every name. tests/v2/test_parser.py's `test_reading_languages_name_the_default_files_under_a_choice`, which pinned the old answer, is rewritten as `test_reading_languages_follow_the_choice`, beside tests for strict, a caller's log, an alternative and a pickled parser.
520520
- **Measured.** tests/v2/test_snapshot.py passes unchanged; `uv run python tools/differential/cldr_agreement.py` reports the CLDR locale view and the Wikidata native locale view matching their records (JA not measured, namedivider absent); frames, `uv run python tools/perf/call_count.py --against 854087be`, the reference row 311.48 to 311.72, all of it the first parse of each word's miss (311.0 on both in a second pass), `_CALL_BASELINE` unchanged and the serial benchmark passing; the first listed-word parse is unchanged in time for the default and for `parser_for(locales.PT)`.
521+
- **2026-11-02 #754, the 2.x translation's lifetime (Derek) — THE TRANSLATION OF 2.x SPELLINGS AND PICKLES IS MIGRATION SCAFFOLDING: IT STAYS IN 3.0, A 2.x `Lexicon` PICKLE LOADING WITH A `DeprecationWarning`, AND IS DELETED IN 3.1.** Sets the lifetime item 6 of the 2026-11-02 #754, the `Lexicon` surface entry left unstated: its old spellings and item 5's 2.x pickle share one translation, `_lexicon`'s `_target_readings`, `_inert_dropped` and `_MARKER_READ_WITH`, `_edits_toward`, `_translation` and `Lexicon._changed`, with `_translated_2x` and `_refusal` built on it, about 500 lines grown across 4i-b's review rounds, and nothing a 3.0 caller writes reaches it.
522+
- **3.0.** A 2.x `Lexicon` state loads as item 5 says and then warns, a `DeprecationWarning` saying it was written by 2.x, that 3.1 will refuse it, and to re-pickle the loaded object under 3.0; it is raised once per `Lexicon` loaded, after the load succeeds, so a `Parser` or locale pack holding one warns once, and a refused state raises without it. `_warn_at_caller` attributes it to the caller's `pickle.loads` line, the C unpickler adding no Python frame between it and `__setstate__`. A pickle written by 3.0, 4i-b's layout included, does not warn. The old-spelling refusals are unchanged: they raise already, with their per-word translation, so no warning is added to them.
523+
- **3.1.** The translation is deleted. An old spelling (`add`/`remove` with a set keyword, `Lexicon(...)` or `dataclasses.replace` with one) raises a short fixed `TypeError` naming the field and pointing at docs/migrate.rst's table of kinds, with no per-word call; a set attribute read keeps its fixed `AttributeError`, which never translated. A 2.x `Lexicon` state is refused with `ValueError` saying to re-pickle under 3.0, as 2.7 refused a 2.6 pickle.
524+
- **Why a minor release.** Neither removal breaks a call that works in 3.0: every old spelling already fails there, and only its message shortens, and a pickle break ships in a minor with a release-log bullet, as 2.6 and 2.7 shipped theirs. A 3.0 release in between gives the one re-pickle that keeps a 2.x pickle.
525+
- Declined: keeping the translation through 3.x (it costs a reader of `_lexicon` 500 lines for a migration done once per pickle and once per call site); warning on the old spellings too (they raise, and a warning beside a raise says nothing more); deleting it in 3.0 (decided against on 2026-11-02 in the `Lexicon` surface entry: the default and the 2.8 fixtures translate back exactly).

‎docs/migrate.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -441,6 +441,13 @@ with the 23 credentials and 13 case masks 3.0 added taken out again, so it reads
441441
``"Jane Doe KC"`` as 2.8.0 did. Build a new ``Parser()`` for 3.0's
442442
vocabulary.
443443

444+
Loading a 2.x pickle is deprecated: it emits a ``DeprecationWarning``,
445+
once for each ``Lexicon`` loaded, and 3.1 will refuse it. Load it under
446+
3.0 and pickle the loaded object again (``pickle.dumps(obj)``); that
447+
pickle loads without a warning, in 3.1 too. The errors a 2.x spelling
448+
raises in 3.0 name the calls that replace it word by word; in 3.1 they
449+
point at the table above instead.
450+
444451
A 2.x call or pickle that took a word out of a set but left it in a
445452
marker over that set (``remove(titles={"air"})``, which left ``air`` in
446453
``title_modifiers``) reads as though the marker went with the word: in

‎docs/release_log.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Release Log
2828
**Breaking Changes**
2929

3030
- **Configure a Lexicon by reading; its set fields are internal.** ``Lexicon.default().remove("dean", kind="given")`` reads ``Dean Smith`` title ``Dean``, the spelling of 2.8.0's ``Lexicon.default().add(titles={"dean"})``, which now raises ``TypeError`` naming that call, as do ``Lexicon(titles=...)`` and ``dataclasses.replace`` with a set field, and reading ``lexicon.titles`` raises ``AttributeError``; ``add(*words, kind=..., **properties)``, ``remove(*words, kind=None)`` and ``readings_of(word)``, returning ``LexiconReading``, take the vocabulary files' kinds, and :doc:`migrate` has a row per 2.8.0 field. A 2.8.0 pickle of a ``Lexicon``, ``Parser`` or ``Locale`` loads where readings compile its sets back, keeping the vocabulary it was pickled with rather than taking 3.0's, and is refused otherwise; a 2.x call or pickle that left a word in a marker set over a set it took the word out of reads as though the marker went with it (after ``remove(titles={"air"})``, ``John Smith, AIR`` gives suffix ``AIR``, given ``AIR`` in 2.8.0). :issue:`754`
31+
- **Deprecate loading a Lexicon, Parser or locale pack pickled by 2.x, which 3.1 refuses.** ``pickle.loads`` of a ``Parser`` pickled by 2.8.0 emits a ``DeprecationWarning``, once for each ``Lexicon`` it holds, and 3.1 will refuse it; re-pickle the loaded object under 3.0 and that pickle loads without one. The errors naming a word-by-word fix for a 2.x ``Lexicon`` spelling keep it in 3.0 and point at :doc:`migrate` in 3.1. :issue:`754`
3132
- **Remove HumanName, Constants and the rest of the 1.x compatibility layer.** ``from nameparser import HumanName`` raises ``ImportError``, where 1.4.0 through 2.8.0 imported it (2.8.0 warning where it was constructed); ``parse("Dr. Juan de la Vega")`` reads the same name, with given ``Juan`` and family ``de la Vega``. ``nameparser.parser``, ``nameparser.util``, ``nameparser.config``'s ``Constants``, ``CONSTANTS`` and managers, and the five word-list names and two module paths 2.2 renamed go too, and a pickled ``HumanName`` or ``Constants`` no longer loads; the word lists in ``nameparser.config`` stay. See :doc:`migrate`. :issue:`357`
3233

3334
**Behavior Changes**

0 commit comments

Comments
 (0)