Define themes as JSON data, loadable from files and packages - #15356

Themes were Python dicts mapping pygments token objects to style strings, built eagerly at IPython.utils.PyColorize import. Adding one meant editing IPython's source — which docs/source/config/details.rst already apologised for:

while currently you need to modify source code to add a theme, it should be possible to load theme from Json, Yaml, or any other declarative file type.

This does that.

Themes are data

Each theme is a JSON file in IPython/utils/themes/. Token types are dotted paths with the leading Token. dropped:

{
  "name": "gruvbox-dark",
  "base": "gruvbox-dark",
  "extra_style": { "Prompt": "#689D6A", "Prompt.Continuation.L1": "#D79921" },
  "symbols": { "top_line": "─", "arrow_body": "─", "arrow_head": "▶" }
}

Theme.from_dict / Theme.from_file do the conversion, resolving tokens only when a theme is actually looked up.

The shipped themes are unchanged. The JSON was generated from the existing definitions rather than transcribed, then all nine themes were diffed against a pre-change snapshot: zero differences.

Three places a theme can come from

1. Bundled — the JSON above.
2. Your IPython directory — drop ~/.ipython/themes/my-theme.json and %colors my-theme works, no code to write. Listed on each lookup, so a file added mid-session is found without restarting.
3. An installed package — via the new ipython.themes entry point group. The entry point is named after the theme; its value says where the JSON lives:

[project.entry-points."ipython.themes"]
solarized-dark = "my_themes"              # my_themes/solarized-dark.json
solarized-light = "my_themes:data"        # my_themes/data/solarized-light.json

A theme package contains no code — JSON files and an empty __init__.py. IPython reads the data and never imports an object from it or calls into it.

Neither external source is consulted until a lookup misses the built-ins, so the ~25 ms importlib.metadata scan and the IPython directory both stay off the startup path. Precedence is bundled → IPython directory → installed package, so %colors linux always means the linux IPython ships.

Validation

A theme is now data that can arrive from elsewhere, so it is checked before it loads and refused with a warning if it fails.

Token paths must match [A-Z]\w* per segment — what pygments itself tests to decide whether an attribute access names a subtoken. Without it, a key like "split.__globals__" or "__class__.__base__" resolved to whatever getattr found and walked off the token graph into the pygments module globals. Not exploitable (attribute reads alone give no call or subscript primitive), but a theme file should be inert by construction rather than by argument.

Symbols are written straight to the terminal, and make_arrow repeats arrow_body besides, so they must be at most 20 printable characters — otherwise a theme could smuggle in an escape sequence and set the window title, move the cursor, or read back the clipboard via OSC 52.

str.isprintable() is the obvious check and is wrong here: it rejects the private use area, where Powerline separators and Nerd Font glyphs live, which is exactly what someone wants an arrow head to be. Rejecting unicode categories Cc, Cf, Cs, Zl, Zp covers what an escape sequence is built from while leaving private use alone; Cf also blocks the bidi overrides. Cn (unassigned) is deliberately not rejected — it would make a symbol's acceptance depend on how old the running Python's unicode database is, since a new emoji is Cn until Python catches up.

Names: a theme is always stored as _theme_filename of its name, both for the file a name is looked up in and for the name a file in the IPython directory is known by. That keeps the two agreeing in either direction, and stops a name reaching outside its package — files(pkg) / "../../../x.json" resolves rather than failing.

Incompatibilities

Neither is relied on anywhere in the tree:

• theme_table is a lazy Mapping rather than a dict. Lookup, iteration and .keys() work as before; it can no longer be mutated with __setitem__.
• The individual themes are no longer module-level objects — use theme_table["linux"] rather than PyColorize.linux_theme.

Notes for review

• Deliberately not a performance change: pygments.token costs 0.56 ms and building all nine themes well under 1 ms. The two expensive imports (pygments.styles at 27 ms, pygments.formatters at 29 ms) were already deferred. This is for extensibility.
• Drive-by: functools.cache on Theme.as_pygments_style / _get_formatter keyed on self, keeping every Theme alive forever. Harmless while themes were nine module singletons; a leak now that anything can build one. Now cached on the instance, with no call sites changed.
• Packaging: pyproject.toml gains "IPython.utils" = ["themes/*.json"]; MANIFEST.in's graft IPython covers the sdist.

Verified end to end against a genuinely pip-installed theme package and a real IPYTHONDIR, not just stubs. Tests are in tests/test_themes.py; the validation guards were checked by mutation — disabling any one of them fails between 3 and 16 tests.