Skip to content

Check n before iter() in itertools.tee() and re-check the iterator - #9518

Merged
youknowone merged 1 commit into
RustPython:mainfrom
jseop-lim:fix/itertools-tee
Nov 1, 2026
Merged

youknowone merged 1 commit into
RustPython:mainfrom
jseop-lim:fix/itertools-tee

Conversation

@jseop-lim

@jseop-lim jseop-lim commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

TeeArgs binds iterable as PyIter, so iter() runs while the arguments are bound, before tee checks n. tee then copies an iterator that has __copy__ directly, and PyItertoolsTee::from_iter stores any other iterator without calling iter() on it. In CPython the argument clinic rejects a negative n, and itertools_tee_impl returns () for zero before calling iter(), then buffers the iterator in a tee object, whose tee_fromiterable calls iter() on it again.

AS-IS

>>> import itertools
>>> itertools.tee(NoneIter(), -1)  # __iter__ returns None
TypeError: NoneIter.__iter__() must return an iterator, not NoneType
>>> itertools.tee(NoneIter(), 0)
TypeError: NoneIter.__iter__() must return an iterator, not NoneType
>>> type(itertools.tee(BadIterable()))  # its iterator's __iter__ returns None
<class 'tuple'>

TO-BE

>>> itertools.tee(NoneIter(), -1)
ValueError: n must be >= 0
>>> itertools.tee(NoneIter(), 0)
()
>>> type(itertools.tee(BadIterable()))
TypeError: BadIterator.__iter__() must return an iterator, not NoneType

(matching CPython's exception types; CPython 3.15 words the first message n cannot be negative)

Changes

  • In crates/vm/src/stdlib/itertools.rs, bind iterable as PyObjectRef. tee checks n, returns an empty tuple for zero, and only then calls iter().
  • Drop the __copy__ shortcut, so every iterator goes through PyItertoolsTee::from_iter, which now calls iter() on the iterator it is given. A tee object is still copied rather than wrapped. An iterator's own __copy__ is no longer called, as in CPython since gh-123884.
  • Extend extra_tests/snippets/stdlib_itertools.py with a negative and a zero n on a broken iterable, an iterator whose __iter__ returns a non-iterator, and an iterator whose __copy__ raises.

Test plan

Built and run on macOS aarch64 against upstream/main 713e14c, using the CI feature set --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls-aws-lc,host_env,doc,ucd_3_2_0, plus threading, compared with CPython 3.15.2.

  • The extended extra_tests/snippets/stdlib_itertools.py fails on an unpatched build of that commit (at tee(NoneIter(), -1)) and passes with the patch and under CPython 3.15.2.
  • rustpython -m test test_itertools: 137 run, 22 skipped, all pass.
  • pytest -k rustpython in extra_tests: 248 pass.
  • cargo fmt --check clean, cargo clippy -p rustpython-vm -- -Dwarnings reports nothing, and the pre-commit hooks pass on both changed files.

I used Claude Code (claude-opus-5-5) to draft the analysis, the patch and this description; I reviewed the change and ran every command above myself.

Summary by CodeRabbit

  • Bug Fixes
    • Improved iterator duplication to accept iterable inputs and handle invalid iterators with a clear error.
    • Requesting zero iterators now returns an empty result without requiring a valid input iterator.
    • Iterator duplication no longer fails solely because an input’s copy operation raises an error.

tee() bound its iterable as an iterator, so iter() ran during argument
binding: a negative n was reported only after __iter__ succeeded, and
n == 0 still called it. tee() also used an iterator's own __copy__
instead of buffering it, and never called iter() on the iterator it
buffered, so an iterator whose __iter__ returns a non-iterator went
unnoticed.

Check n first and return an empty tuple for zero, then always buffer
the iterator in a tee object, which calls iter() on it again.

Assisted-by: Claude Code:claude-opus-5-5
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T04:32:15.498254Z c2b8a24 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 88f3733 and c2b8a24.

Files selected for processing: 2
  • crates/vm/src/stdlib/itertools.rs
  • extra_tests/snippets/stdlib_itertools.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Walkthrough

itertools.tee now checks the requested count before obtaining an iterator. For positive counts, it obtains an iterator and validates it again during tee construction. Tests cover negative and zero counts, invalid iterators, and iterators whose __copy__ raises an exception.

Changes: tee construction and validation

Layer / File(s) Summary
Input handling and result creation
crates/vm/src/stdlib/itertools.rs, extra_tests/snippets/stdlib_itertools.py
tee accepts a Python object, returns an empty tuple for n == 0, and obtains an iterator for positive counts. PyItertoolsTee::from_iter validates its input with get_iter. Tests cover count handling, invalid iterators, and __copy__ failures.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Possibly related PRs

  • RustPython/RustPython#9322: It changes shared tee buffer storage, cursor advancement, and tee copying in the same implementation.

Suggested reviewers: youknowone

Merge Risk: Minimal · up to c2b8a

No concrete merge-blocking issue is established for the tee changes; mergeable subject to normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main changes: validating n before calling iter() and re-checking the iterator in itertools.tee().
Linked Issues check Passed The changes satisfy the coding requirements in direct issue #9512. TeeArgs now keeps the iterable as PyObjectRef, so tee() can validate negative n before iter(). The implementation returns t…
Out of Scope Changes check Passed The diff stays within issue #9512. The Rust changes implement the required itertools.tee() behavior. The Python changes add focused regression tests for those behaviors. No unrelated production beha…
Full details: Linked Issues check

Explanation

The changes satisfy the coding requirements in direct issue #9512. TeeArgs now keeps the iterable as PyObjectRef, so tee() can validate negative n before iter(). The implementation returns the empty tuple for n == 0 before iterator conversion. For positive n, it calls get_iter() on the iterable and again in PyItertoolsTee::from_iter before buffering. It removes the direct input __copy__ shortcut while retaining copying for tee objects. The added tests cover negative and zero counts, invalid iterator conversion, and an iterator whose __copy__ raises.

Full details: Out of Scope Changes check

Explanation

The diff stays within issue #9512. The Rust changes implement the required itertools.tee() behavior. The Python changes add focused regression tests for those behaviors. No unrelated production behavior or unrelated tests are changed.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codspeed

codspeed Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing jseop-lim:fix/itertools-tee (c2b8a24) with main (88f3733)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@youknowone
youknowone merged commit 13e32c8 into RustPython:main Nov 1, 2026
21 checks passed
@jseop-lim
jseop-lim deleted the fix/itertools-tee branch November 1, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

itertools.tee() calls iter() before checking n, and never calls it on the iterator it buffers

2 participants