Skip to content

Fix RuntimeError when pretty-printing a dict mutated during iteration - #15472

Open
bobby-nandigam wants to merge 1 commit into
ipython:mainfrom
bobby-nandigam:fix/pretty-dict-mutation-during-iteration
Open

bobby-nandigam wants to merge 1 commit into
ipython:mainfrom
bobby-nandigam:fix/pretty-dict-mutation-during-iteration

Conversation

@bobby-nandigam

Copy link
Copy Markdown

Summary

Fixes #14541.

Pretty-printing a mapping that mutates while it is being formatted raised:

RuntimeError: dictionary changed size during iteration

The reported trigger is vars(posix) on its first call: a module's
__dict__ populates lazily the first time its attributes are accessed, so it
grows while the pretty-printer is iterating it (which is why it works on every
subsequent call, or after completion has already touched the attributes).

Root cause

_dict_pprinter_factory in IPython/lib/pretty.py iterated the mapping's
live key view and looked each value up again:

keys = obj.keys()
for idx, key in p._enumerate(keys):
    ...
    p.pretty(obj[key])

If anything mutates obj during the loop, iterating the live view raises
RuntimeError.

Fix

Snapshot the items once, up front, and iterate that snapshot:

items = list(obj.items())
for idx, (key, value) in p._enumerate(items):
    ...
    p.pretty(value)

The live mapping is now read exactly once, before any value is formatted, so a
mutation triggered during formatting can no longer break iteration. Capturing
(key, value) pairs also removes the second live obj[key] lookup. Behaviour
for ordinary, non-mutating mappings (including dict, os.environ, and
types.MappingProxyType, which share this factory) is unchanged.

Testing

  • Added test_dict_mutated_while_pretty_printing, which formats a dict that
    grows while one of its values is being pretty-printed. It reproduces the
    exact RuntimeError on main and passes with this change.
  • pytest tests/test_pretty.py — all tests pass (54).
  • ruff check reports no new findings on the changed lines.

Notes

Added a docs/source/whatsnew/pr/ fragment describing the fix.

The dict pretty-printer iterated a live key view (obj.keys()) and looked
up each value with obj[key]. A mapping that grows while being formatted --
for example a module's __dict__ that populates lazily on first attribute
access, as happens with vars(posix) on its first call -- therefore raised
"RuntimeError: dictionary changed size during iteration".

Snapshot the mapping's items with list(obj.items()) before iterating so the
live mapping is read exactly once, up front. This also removes the second
live lookup per key. Behaviour for ordinary, non-mutating mappings is
unchanged.

Add a regression test that formats a dict which grows while one of its
values is being pretty-printed.

Fixes ipython#14541

@raidos23 raidos23 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we limit the snapshot to the number of items that can actually be formatted?

Currently, list(obj.items()) materializes the entire mapping before p._enumerate() applies the max_seq_length limit.

For very large dictionaries, this can result in unnecessary memory consumption compared to the previous implementation.

One possible approach would be to use itertools.islice() to snapshot only the required number of items, plus one extra item to preserve the truncation indicator. When max_seq_length is disabled, the full snapshot could still be retained

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RuntimeError on first call to vars(posix)

2 participants