Divide a bracketed clause's words at a glued comma (#1268) - #1339

Summary

A bracketed clause of credentials glued at a comma reads as credentials:

parse("John Smith (MD,PhD)")      suffix 'MD,PhD'   (every release a nickname: 'MD PhD' in 2.0.0–2.4.0, 'MD,PhD' otherwise)
parse("John Smith (Jr.,MD)")      suffix 'Jr.,MD'   unchanged (3.0.0 on)
parse("Jane (Janey,Jo) Smith")    nickname 'Janey,Jo'   unchanged
parse("John Smith (Dr.，PhD)")    nickname          unchanged: a title glued to a credential
parse("John Smith (Ma,PhD)")      nickname          unchanged: a word that may be a name
parse("John Smith (MD,,PhD)")     nickname          unchanged: a doubled comma

Re-scoped in the fourth round. The first design ended a clause's word at a glued comma in tokenize, in every clause, nicknames included. Three review rounds then compensated for it without converging (a rejoin in Parser.revise, guards for degenerate shapes). That design is gone:

• tokenize, extract, _vocab and _parser.py (revise) are master's;
• the change is in post_rules' _set_apart_clauses alone, where rules/S1 is decided for good.

How

• A clause's token holding a single comma glued between two words is read as those words, for S1's tests (_post_rules._glued_kind).
• Where every word takes one kind (all credentials or generations, or all titles, a title/suffix dual taking its neighbours'), the token is set apart whole and keeps its comma as written.
• Any other token keeps the base's reading.
• A clause S1 does not set apart yields the base's tokens, tags and text.

Why the token is not divided:

• S1 is decided twice: provisionally in extract, before tokens exist, and for good in post_rules, two stages after tokenize.
• Each declines clauses the other takes: post_rules declines Justin (JP,PhD) Stearns, John (MD,PhD.) Smith and (Dr.,Prof.), which extract approved, and sets apart (Ma, PhD), which extract refused.
• So tokenize can't divide only the clauses S1 sets apart.
• Dividing in post_rules would insert tokens after the pieces, the segments and the stages' reports already index them.

Decision for the maintainer

• Chosen: the glued word is set apart whole and reads as written: suffix MD,PhD, not the spaced twin's MD, PhD.
• What it costs:
  • A title glued to a credential ((Dr.，PhD), the issue's fullwidth example) stays a nickname, since one token can't take two roles.
  • A word that may be a name ((Ma,PhD), (MD/PhD,MBA)) stays a nickname too.
  • revise(suffix="MD,PhD") gives MD, PhD: a bare value has no clause, so the comma divides. This is recorded as a round-trip limit beside the glued-word one.
• Alternative: divide the token, so the reading matches the spaced twin and the mixed word reads as title plus suffix. That needs a token split in post_rules with every token index remapped, or a division in tokenize that also reaches clauses S1 leaves as nicknames. That is the design that didn't converge.
• Also possible: park the PR. The shape is in 3 ORCID names and in no real ClinicalTrials.gov name.
• The issue's open questions: a nickname keeps its comma as written (the first reading). The semicolon is left alone: a spaced one in a clause already reads as a nickname.

Measurements (against 8437022, a git archive, __file__ asserted)

• revise: 18 values × 3 spellings × 7 fields, compared on fields, initials() and capitalized(): 0 of 378 differ from the base.
• Shapes the reviews found: all read as at the base: doubled commas, a comma before or behind a dropped separator, (,,MD), (,,Jr.) between and after the name, (MD,,PhD), and comma-free clauses holding a separator.
• What moves:
  • John Smith (MD,PhD);
  • the same text under bracket maiden_delimiters (maiden → suffix, S1 taking a maiden pair's clause first);
  • two ORCID names: Dr. Thierry Ingabire, (MD,PhD) and Dr. Md Safiul Alam Bhuiyan (DVM.MS,PhD).
• Committed snapshot: no existing name moves; the new rule and case examples are added.
• snapshot.py --local ctgov-officials --base: matches the base. match_diff: 0 of 654,864 rows move.
• Frames:
  • A plain name and a clause with no glued comma pay nothing: John Smith 126, John "Jack" Smith 198, John Smith (MD, PhD) 247, each as at the base.
  • A clause with a glued comma pays the word's reading: (MD,PhD) 196 → 214, (Janey,Jo) 198 → 199.

Docs

• rules/S1: the comma bullet, an example, and an Accepted bullet for the mixed word, a member and a doubled comma.
• decisions/S1: a dated Should John Smith (MD,PhD) read the credentials when the comma has no space beside it? It gives nickname MD,PhD today #1268 entry, with the declined design and why.
• rules/N1 and decisions/N1: back to master.
• A release-log bullet.
• nameparser/AGENTS.md's spacing sweep records why the division couldn't live in tokenize.

Checks

On 8ea9f50: uv run pytest (45,185 passed), uv run pytest -n0 -m serial (62 passed), uv run mypy, uv run ruff check, sphinx-build -b doctest, all clean.

Merge note

Merged with master f03856f (#1340, #1341). PR 1338 touches the same release-log place and the snapshot files; whichever lands second needs a master merge and snapshot.py --record.

Closes #1268

🤖 Generated with Claude Code