gh-142183: Cache one datachunk per tstate to prevent alloc/dealloc thrashing by Yhg1s · Pull Request #145789 · python/cpython

Yhg1s · 2026-03-11T00:10:18Z

Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary.

Issue: Calls across stack chunks perform badly #142183

repeatedly hitting the same call depth at exactly the wrong boundary.

Yhg1s · 2026-03-11T00:35:06Z

Just to be clear: this is effectively a freelist of 1, and there's still an easily crafted (but much less likely in reality, I would argue) case where two (or more) stack chunks are repeatedly allocated and deallocated. That requires a much larger chain of calls -- or much larger functions -- so it's not as pronounced, but crafting code to hit that exact case isn't hard. It shows a ~15% penalty for being at just the wrong stack depth, compared to 35+% for the single chunk case. I considered making the cached chunk a freelist (which would be easy since they're already a linked list) but this would mean keeping all datastack chunks of a thread alive for the entire duration of a thread, which might not be a good idea. Caching a single chunk seems like a reasonable compromise.

Here's some benchmark results using the repro I provided in the issue, run on a not particularly quiet machine so the results are a little noisy. 55 is the stack depth level that triggers the bad case, 56 is one level deeper (so slightly more work) and avoids it:

% hyperfine --warmup 3 './base/python repro.py' './fixed/python repro.py'
Benchmark 1: ./base/python repro.py
  Time (mean ± σ):      24.1 ms ±   3.8 ms    [User: 15.9 ms, System: 7.7 ms]
  Range (min … max):    20.9 ms …  41.8 ms    112 runs

Benchmark 2: ./fixed/python repro.py
  Time (mean ± σ):      18.4 ms ±   2.0 ms    [User: 14.8 ms, System: 3.3 ms]
  Range (min … max):    16.4 ms …  28.3 ms    150 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ./fixed/python repro.py ran
    1.31 ± 0.25 times faster than ./base/python repro.py

%  hyperfine --warmup 3 './base/python repro.py 55' './base/python repro.py 56'
Benchmark 1: ./base/python repro.py 55
  Time (mean ± σ):      21.6 ms ±   2.4 ms    [User: 14.6 ms, System: 6.7 ms]
  Range (min … max):    19.5 ms …  31.5 ms    128 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./base/python repro.py 56
  Time (mean ± σ):      16.7 ms ±   1.3 ms    [User: 13.7 ms, System: 2.8 ms]
  Range (min … max):    15.3 ms …  25.1 ms    165 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ./base/python repro.py 56 ran
    1.30 ± 0.17 times faster than ./base/python repro.py 55

% hyperfine --warmup 3 './fixed/python repro.py 55' './fixed/python repro.py 56'
Benchmark 1: ./fixed/python repro.py 55
  Time (mean ± σ):      17.1 ms ±   1.5 ms    [User: 13.8 ms, System: 3.1 ms]
  Range (min … max):    15.6 ms …  24.6 ms    164 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./fixed/python repro.py 56
  Time (mean ± σ):      17.1 ms ±   2.0 ms    [User: 13.8 ms, System: 3.1 ms]
  Range (min … max):    15.8 ms …  26.4 ms    170 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ./fixed/python repro.py 55 ran
    1.00 ± 0.15 times faster than ./fixed/python repro.py 56

markshannon

Looks good. Thanks for fixing this.

Do you want to backport this to 3.14 and maybe 3.13?

Note:
Although this is a solid fix for 3.14 and 31.3, we'll probably want to use a resizable stack for 3.15+ to avoid deopts in the SAI and JIT when operating at the edge of a chunk

Yhg1s · 2026-03-11T14:45:57Z

I think we should backport, yes, unless @hugovk thinks it's a bad idea.

miss-islington-app · 2026-03-11T14:46:21Z

Thanks @Yhg1s for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

…loc thrashing (pythonGH-145789) Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. --------- (cherry picked from commit 706fd4e) Co-authored-by: T. Wouters <thomas@python.org> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

bedevere-app · 2026-03-11T14:46:34Z

GH-145828 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2026-03-11T14:46:39Z

GH-145829 is a backport of this pull request to the 3.13 branch.

…lloc thrashing (GH-145789) (#145828) Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. Move new _ts member to the end to not mess up remote debuggers' ideas of the struct's layout. (The struct is only created by the runtime, and the new field only used by the runtime, so it should be safe.) (cherry picked from commit 706fd4e) Co-authored-by: T. Wouters <thomas@python.org> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

…loc thrashing (pythonGH-145789) (python#145828) Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. Move new _ts member to the end to not mess up remote debuggers' ideas of the struct's layout. (The struct is only created by the runtime, and the new field only used by the runtime, so it should be safe.) (cherry picked from commit 706fd4e) (cherry picked from commit 19cbcc0) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

bedevere-app · 2026-03-18T16:25:19Z

GH-146123 is a backport of this pull request to the 3.13 branch.

Summary: Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. Upstream issue: python/cpython#142183 Upstream PR: python/cpython#145789 Reviewed By: itamaro Differential Revision: D96052502 fbshipit-source-id: 883f6148568b6cca5a71dd7d754eb206fbb56f90

…lloc thrashing (GH-145789) (#146123) * gh-142183: Cache one datachunk per tstate to prevent alloc/dealloc thrashing (GH-145789) (#145828) Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. Move new _ts member to the end to not mess up remote debuggers' ideas of the struct's layout. (The struct is only created by the runtime, and the new field only used by the runtime, so it should be safe.) (cherry picked from commit 706fd4e) (cherry picked from commit 19cbcc0) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>

…loc thrashing (python#145789) Cache one datachunk per tstate to prevent alloc/dealloc thrashing when repeatedly hitting the same call depth at exactly the wrong boundary. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

CPython 3.11+ keeps frames in per-thread data-stack chunks. When a hot call sits at a chunk boundary, every call maps a fresh chunk and every return unmaps it (gh-142183). On GB200 each munmap also pays a Grace SMMU invalidation once NCCL has peer-mapped memory, so the churn costs milliseconds per executor step: MiniMax-M3 piecewise CUDA graph prefill steps spent about half their host time in munmap, all on the executor thread. CPython fixed this in 3.13.13 and 3.14.4; 3.12 will not get the fix. Add run_with_frame_stack_reserve() and call PyExecutor.event_loop through it, so the loop starts near the beginning of a large data-stack chunk and ordinary call depths never reach a boundary. The reserve is applied only on affected CPython versions and can be disabled with TLLM_DISABLE_FRAME_STACK_RESERVE=1. Cover the version gate, the disable switch, the churn itself and the executor wiring with cpu_only unit tests. Background: - python/cpython#142183 (the issue) - python/cpython#145789 (the fix; backports #145829 / #146123 for 3.13, #145828 for 3.14) - python/cpython#88756 (3.11 change that moved frames onto the data stack) - sgl-project/sglang-torchtpu#1037 (same technique) Signed-off-by: Dan Hansen <218031328+dhansen-nvidia@users.noreply.github.com>

Cache one datachunk per tstate to prevent alloc/dealloc thrashing when

8183701

repeatedly hitting the same call depth at exactly the wrong boundary.

Yhg1s requested a review from markshannon March 11, 2026 00:10

Yhg1s requested review from FFY00, ZeroIntensity and ericsnowcurrently as code owners March 11, 2026 00:10

bedevere-app Bot added the awaiting core review label Mar 11, 2026

bedevere-app Bot mentioned this pull request Mar 11, 2026

Calls across stack chunks perform badly #142183

Open

📜🤖 Added by blurb_it.

56a09cb

Avoid sign comparison issues.

88cef81

markshannon approved these changes Mar 11, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Mar 11, 2026

Yhg1s added needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes labels Mar 11, 2026

Yhg1s merged commit 706fd4e into python:main Mar 11, 2026
58 checks passed

bedevere-app Bot removed the awaiting merge label Mar 11, 2026

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Mar 11, 2026

bedevere-app Bot removed the needs backport to 3.13 only security fixes label Mar 11, 2026

Yhg1s deleted the cache-datachunk branch August 13, 2026 12:51

peterdsharpe mentioned this pull request Oct 19, 2026

[Performance] test_stack_nested in the benchmark suite takes 1.6 times as long since #2876, though torch.stack got faster pytorch/tensordict#3061

Closed

Navigation Menu

Repository navigation

gh-142183: Cache one datachunk per tstate to prevent alloc/dealloc thrashing - #145789

gh-142183: Cache one datachunk per tstate to prevent alloc/dealloc thrashing#145789
Yhg1s merged 3 commits into
python:mainfrom
Yhg1s:cache-datachunk

Yhg1s commented Mar 11, 2026 •

edited by bedevere-app Bot

Loading

Uh oh!

Yhg1s commented Mar 11, 2026

Uh oh!

markshannon left a comment

Uh oh!

Yhg1s commented Mar 11, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Yhg1s commented Mar 11, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yhg1s commented Mar 11, 2026

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Yhg1s commented Mar 11, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 11, 2026

Uh oh!

bedevere-app Bot commented Mar 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Yhg1s commented Mar 11, 2026 •

edited by bedevere-app Bot

Loading