Lazily import many imports to speed up startup. - #15333

…attr__`

`IPython/__init__.py` eagerly imported these three names, and with them
`IPython.terminal.embed` (the whole terminal / prompt_toolkit stack, by
far the most expensive of the top-level imports),
`IPython.core.application` (traitlets' application machinery plus the
crash handler) and `IPython.core.getipython`. Move them behind PEP 562
module `__getattr__`.

`IPython.Application` turns out to be nothing but a re-export of
`traitlets.config.application.Application`, and nothing in the Jupyter
ecosystem imports it from here -- ipykernel imports
`BaseIPythonApplication` from `IPython.core.application` directly. It
therefore warns (`DeprecationWarning`) when reached through this module,
both to point at traitlets and to surface code that relies on `import
IPython` transitively importing `IPython.core.application`. `embed` and
`get_ipython` stay silent: both are widely and legitimately used from
here (ipywidgets, matplotlib and comm all do `from IPython import
get_ipython`).

`Application` is deliberately left out of `__dir__`, so that tools which
walk `dir(IPython)` and getattr() the result -- our own module completer
among them -- do not trip the warning without any code asking for the
name. Nothing is cached in `globals()` either, so the warning keeps
firing rather than only on first access, and these names never turn into
plain module attributes that later code could mistake for eagerly
imported ones.

Deferring `get_ipython` saves no import time (`IPython.core.getipython`
is pulled in anyway via `IPython.core.magic`); it is deferred for
consistency with the other two.

The remaining top-level imports are left eager, since downstream
(pyflyby at least) relies on their transitive side effects.

`IPython.core.application` imported `logging` at module level only to
spell `logging.DEBUG` / `logging.CRITICAL` in the `--debug` / `--quiet`
flag definitions. Introduce `LOGGING_DEBUG` and `LOGGING_CRITICAL`
module constants holding those two values, with a comment explaining
that they exist to keep `logging` off the startup path, and drop the
import; the flag help strings still name the levels.

The levels are part of `logging`'s documented public API and cannot
change, but `tests/test_application.py::test_logging_level_constants`
asserts that the copies -- and the flag definitions built from them --
do not drift from `logging` anyway.

`IPython.core.tips` imported argcomplete purely to decide whether one tip
out of ~30 was eligible to be shown. That import costs ~3 ms and 8
modules on every startup, for a string that is usually not even picked.

`importlib.util.find_spec` answers the same question by looking the name
up on `sys.path`, without executing the package. Using
`importlib.metadata` instead would be a step backwards: querying it costs
more than the import it replaces (27 ms / 97 modules, versus 19 ms / 61
for `import argcomplete`, versus 1.6 ms / 9 for `find_spec` in a clean
interpreter), and it drags `importlib.metadata` -> `zipfile` -> `shutil`
onto the startup path, which we are trying to get off it.

`random.choice` moves into `pick_tip` while we are here. Both names were
monkeypatched by tests through this module; those now patch the
definition site, and the argcomplete check gains a test for the
not-installed branch, which nothing covered before.

In a clean interpreter `import IPython.terminal.ipapp` goes from 506 to
496 modules.

A batch of imports that only matter once a particular feature is used,
moved to their use sites:

 - `docrepr.sphinxify` in `core/interactiveshell.py`: the whole
   try/except-ImportError dance at module level becomes a `sphinxify()`
   that imports docrepr (and the sphinx it drags in) when the provisional
   rich-docstring feature is actually used;
 - `stack_data` in `core/tbtools.py`, `core/doctb.py` and
   `core/ultratb.py`, needed only while a traceback is being formatted;
 - `asyncio` (and `asyncio.exceptions`) in `core/async_helpers.py` and
   `terminal/interactiveshell.py`;
 - `pydoc` and `urllib.request` in `core/magics/code.py`, `pstats` in
   `core/magics/execution.py`, `subprocess` in `core/magics/script.py`;
 - the debugger imports (`IPython.core.debugger`, `pdb.Restart`) in
   `core/interactiveshell.py` and `terminal/interactiveshell.py`.

oinspect backs `?`/`??` object introspection (Inspector, OInfo) but was
imported unconditionally at module load time from several places:
interactiveshell.py (both as a direct import and as the eager default
for the inspector_class trait), magic.py (arg_err's docstring
fallback), magics/code.py and magics/osm.py (rare error/edit-target
paths), and splitinput.py (a deprecated helper).

Deferred all of these to where they're actually used: runtime imports
inside the relevant methods, `inspector_class` now resolves its klass
lazily via traitlets' dotted-string form (resolved when InteractiveShell
is instantiated, not when the class body executes), and the handful of
`OInfo` return-type annotations are now quoted so they don't require an
eager import merely to define the function signature.

Together with the terminal.embed laziness now in IPython/__init__.py,
this fully removes IPython.core.oinspect (and IPython.core.magic no
longer eagerly pulls it either) from a bare `import IPython`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

`IPython.utils.PyColorize` is imported at startup by five modules
(`core/tbtools.py`, `core/completer.py`, `core/interactiveshell.py`,
`core/oinspect.py`, `core/debugger.py`), and it in turn pulled in two
expensive pygments entry points at import time:

 - `pygments.formatters.terminal256`, which drags in the whole
   `pygments.formatters` package (9.7 ms), needed only by
   `Theme._get_formatter`;
 - `pygments.styles`, which drags in the pygments plugin machinery and
   with it `importlib.metadata` -> `zipfile` -> `shutil` (8.9 ms), needed
   only to resolve a theme's base style.

Both move to their use sites, in `PyColorize` itself and in the two other
places that used them directly (`core/doctb.py` and
`terminal/interactiveshell.py`). `core/ultratb.py` and `core/doctb.py`
also stop importing `PyColorize` at module level: `theme_table` moves
into the methods that format tracebacks, and `Theme`/`TokenStream` under
`TYPE_CHECKING`, which is free since both modules use
`from __future__ import annotations`.

`pygments.styles` and `pygments.formatters` no longer appear at startup
at all.

`IPythonPTLexer.__init__` built a `PygmentsLexer` for every supported
`%%magic` language up front, which imports the pygments submodule
backing each of them (bash, html, javascript, perl, ruby, latex, ...) on
every terminal startup. Most sessions only ever highlight Python.

Wrap them in a small `_LazyPygmentsLexer` that resolves the pygments
class and builds the real lexer on first `lex_document` call instead.

None of these is needed to start a shell, and all of them sit on the
startup path:

 - `subprocess` in `utils/sysinfo.py` (only to ask git for a commit hash
   when running from a checkout) and in `utils/_process_common.py`. The
   latter needed two changes beyond moving the import: `stderr` defaulted
   to `subprocess.PIPE`, a default argument evaluated at definition time,
   so it becomes `None` resolved in the body; and the
   `Callable[[subprocess.Popen[bytes]], _T]` annotation forced the import
   until the module gained `from __future__ import annotations`;
 - `shutil` in `core/application.py` (copying the profile README),
   `core/profiledir.py` (staging config files), `utils/process.py`
   (`shutil.which`) and `utils/path.py` (`link_or_copy`'s fallback);
 - `random` in `utils/path.py`, `tempfile` in `utils/io.py` and
   `paths.py`.

`utils/path.py` also drops `from IPython.utils.process import system`,
which had no remaining users.

Imports on the startup path that only matter on cold paths:

 - `core/interactiveshell.py`: `shutil` (temp-dir cleanup at shutdown),
   `bdb` (the `except bdb.BdbQuit` in `run_code`), `tempfile`
   (`mktempfile`), `traceback` (exception formatting), `subprocess` and
   `subprocess.CalledProcessError` (`system_raw`/`getoutput`), and
   `logging.error` (the `%debug` guard);
 - `core/hooks.py` and `core/page.py`: `subprocess`, for spawning an
   editor and a pager respectively. On POSIX this is worth more than it
   looks, since stdlib `subprocess` probes for the Windows-only `msvcrt`
   module on the way in;
 - `core/page.py`: `tempfile` and `IPython.utils.process.system`;
 - `core/compilerop.py`: `hashlib`, used only by `code_name`;
 - `core/magic.py`: `..utils.process.arg_split`, used only by
   `parse_options`;
 - `core/alias.py`: `logging.error`, used only when an alias definition
   fails;
 - `utils/terminal.py`: `shutil.get_terminal_size`.

`core/magic.py` also drops its unused `from logging import error`.

Two of these names were monkeypatched by tests through the module that
imported them (`page.system`, `page.subprocess`,
`terminal._get_terminal_size`), which stops existing once the import
happens at call time; those tests now patch the definition site, which is
what they meant either way.

`core/ultratb.py`, `core/doctb.py` and `core/tbtools.py` are all imported
at startup, and all three only need these while an exception is actually
being rendered:

 - `traceback`, whose sole runtime user is `ListTB._extract_tb`; the
   remaining `traceback.StackSummary` reference is a return annotation,
   free now that the module has `from __future__ import annotations`;
 - `inspect`, used by `format_record` (`getargvalues`,
   `formatargvalues`) and `get_records` (`getmodule`, `getsourcelines`).

`inspect` stays on the startup path regardless -- `traitlets.traitlets`
imports it -- but IPython no longer has its own reason to.

`IPython.core.history` is on the startup path (via `core.displayhook`),
but a session that never reads or writes history never needs sqlite3,
which costs ~8 ms and 23 modules cold.

Three things kept the import eager, each handled here:

 - `sqlite3.register_converter("timestamp", ...)`, a global registration
   that has to happen before any connection using `PARSE_DECLTYPES`. It
   moves into a cached `_sqlite3()` helper that imports the module and
   registers the converter exactly once, and every use site goes through
   that helper;
 - `enabled = Bool(sqlite3_found, ...)`, whose default was evaluated when
   the class was created. It becomes a `@default("enabled")` dynamic
   default, so availability is decided on first access;
 - `DatabaseError` / `OperationalError` in `except` clauses, previously
   backed by dummy classes when sqlite3 was missing. They become
   `_db_errors()` / `_operational_error()`, returning the real exception
   types, or an empty tuple when sqlite3 is unavailable -- an empty tuple
   in an `except` clause never matches, which is right when there is no
   database to fail.

The module is still named in annotations, so it is imported under
`TYPE_CHECKING` for the type checker's benefit; that costs nothing at
runtime. The `connect()` call loses its `type: ignore[call-overload]`,
which is unused now that it goes through the untyped helper.

Availability is still decided by attempting the import, and by catching
`ImportError` rather than just `ModuleNotFoundError`. `sqlite3` is a
pure-Python package wrapping the `_sqlite3` extension, so it is present
on disk even on interpreters built without sqlite3 support -- a
`find_spec("sqlite3")` check would wrongly report it as available, and
the extension can also be present but fail to load.

In a clean interpreter `import IPython.terminal.ipapp` goes from 496 to
493 modules.

`from collections.abc import Callable, Iterator` and
`from weakref import ReferenceType` sat two thirds of the way down the
file, between `os.register_at_fork(...)` and the `hold()` helper that
uses them. Nothing depends on them being imported late.

`ip.db` was constructed during shell initialization, which imported
`IPython.external.pickleshare` (and `pickle` underneath it, 2.1 ms
together) and created the database directory, whether or not the session
ever touched it. Only `%store`, `%bookmark` and the module-completion
cache use it.

Deferring the import alone would not have helped, since the class was
instantiated at startup regardless, so `db` becomes a lazy property
instead. The setter is kept because plenty of code, in IPython and
downstream, assigns to `ip.db` -- the module completer even checks for a
`_mock` attribute on it.

In a clean interpreter `import IPython.terminal.ipapp` goes from 493 to
491 modules.

`IPython.core.logger.Logger` writes the `%logstart` session transcript;
sessions that never start one never need it, but it was imported and
instantiated during shell initialization. Like `db`, it becomes a lazy
property with a setter.

Its module also carried, at import time, a global mutation belonging to
something else entirely:

    # prevent jedi/parso's debug messages pipe into interactiveshell
    logging.getLogger("parso").setLevel(logging.WARNING)

That has nothing to do with session transcripts, and it only ever ran
because `core.logger` happened to be imported eagerly -- deferring the
import silently stops parso being silenced. Move it to `_get_jedi()` in
`core/completer.py`, where jedi is imported and configured, which is
still before any parso record can be emitted since parso is only reached
through jedi. The `logging` import goes with it; `core/logger.py` had no
other use for it.

In a clean interpreter `import IPython.terminal.ipapp` goes from 489 to
488 modules.

`%%script` only raises it when a cell exits non-zero under
`--raise-error`, so the import moves into `shebang`, next to the other
imports it already defers.

This no longer changes what is imported at startup: `subprocess` is now
pulled in by `asyncio.base_events`, via the prompt_toolkit import in
`terminal/interactiveshell.py`, which an interactive terminal needs
anyway.

`urlencode` has one caller, `%pastebin`, in the same function that
already defers `urllib.request`. Importing it at module level cost 2.5 ms
on every startup (6.7 ms and 30 modules in a cold interpreter, since
`urllib.parse` pulls `ipaddress`).

In a clean interpreter `import IPython.terminal.ipapp` goes from 485 to
482 modules; `urllib.parse` and `ipaddress` no longer appear at startup.

…e completer

`IPython.core.completer` is imported at startup, but none of these is
needed until a completion is actually requested: `uuid` names a profiler
output file, `unicodedata` backs `\GREEK SMALL LETTER ...` completion,
`IPython.core.guarded_eval` evaluates expressions for attribute and dict
key completion, and `IPython.core.latex_symbols` backs `\alpha`
completion. Every use is inside a function body, so they move to their
use sites.

Note the module's own body is not the cost it can appear to be: a
`-X importtime` run over a stale `__pycache__` attributes byte-compilation
of the whole file to it (12 ms here); warm it is under 2 ms. Nor is
`find_spec("jedi")`, which is 0.05 ms.

In a clean interpreter `import IPython.terminal.ipapp` goes from 482 to
479 modules. `unicodedata` and `guarded_eval` remain, now reached via
`terminal/ptutils.py` and `terminal/shortcuts/filters.py`.

This is al alwasy part of improving the startup time, and the import
time of IPython and ipykernel

`clock` and `clock2` are only called by `%time`, `%timeit` and `%run -t`,
so the import moves into those three methods. It pulls `resource` with
it, which nothing else on the startup path wants.

In a clean interpreter `import IPython.terminal.ipapp` goes from 479 to
476 modules.

Both uses are in interrupt handling -- sending SIGINT to a `%%script`
child on Ctrl-C, and to any background processes at cleanup -- so the
import moves to those two sites.

`signal` remains in `sys.modules` at startup regardless, since asyncio
imports it via the prompt_toolkit stack; this only removes IPython's own
module-level import.

Whether the terminal speaks the kitty graphics protocol is guessed by
walking up the process tree looking for a known terminal name. The guess
can be wrong -- inside tmux, in a container, or for an emulator that is
not on the list -- and it is not free: it imports psutil and inspects the
process tree on every startup that has a tty.

`IPYTHON_KITTY_GRAPHICS=1` or `=0` now states the answer outright, and
short-circuits before any of that work. On a tty that takes
`import IPython.terminal.ipapp` from 494 to 488 modules, with psutil no
longer imported at all.

Accepted values are `1`/`true` and `0`/`false`, case-insensitive; unset
or empty keeps autodetecting. An unrecognised value warns and falls back
to autodetection rather than being treated as false, so that a typo
cannot silently disable graphics.

`_LazyPygmentsLexer` already waited until something was actually
highlighted before building a lexer, but the module it looks the classes
up in was still imported at startup -- and that import is the expensive
part: `pygments.lexers` pulls in the pygments plugin machinery, and with
it `importlib.metadata` and `zipfile`.

Move it next to the `getattr` that needs it, in `lex_document`.

In a clean interpreter `import IPython.terminal.ipapp` goes from 476 to
462 modules; `pygments.lexers`, `pygments.plugin`, `importlib.metadata`
and `zipfile` all leave the startup path.

`pylight` is the only user of `highlight`, `PythonLexer` and
`HtmlFormatter`, and it only runs when a docstring is rendered as HTML.
Importing them at module level pulled `pygments.lexers` and
`pygments.formatters`, and with them the pygments plugin machinery,
`importlib.metadata` and `zipfile`.

This is no longer a startup cost -- `core/oinspect.py` itself was taken
off the startup path earlier in this series -- but it is paid on the
first `?`, which is interactive latency: importing `oinspect` now pulls 4
modules instead of 30.

In particular this avoig importing all the machinery in History and
sqlite 3, especially when log_output is not enabled