Repository navigation
Add --stagger <duration> to space out branch pushes and avoid duplicate CI runs #466
Description
The problem in one paragraph
gh stack sync pushes every branch in the stack at once, in a single atomic push. When that happens, GitHub sometimes sends two or three "the PR changed" events for the same commit instead of one. Each extra event starts an extra CI run. Those extra runs get cancelled, but GitHub keeps every check it has ever posted for a commit — so the cancelled ones stay on the pull request's Checks tab forever, showing red. Reviewers open the PR, see red, and think it is broken when it is not.
Adding a short pause between pushes makes the problem go away completely. I measured this, and about 3 seconds was enough.
Why the extra events happen
Every PR in a stack sits on top of the one below it. So when the whole stack is pushed at once, a middle PR has two things change at the same moment: its own branch, and the branch it is based on.
GitHub appears to process those two changes in parallel, and both end up announcing "this PR's commits changed." You get a duplicate event for a commit that only changed once.
The bottom PR of a stack never has this problem, because its base is the trunk and the trunk is not part of the push. Every PR above the bottom is affected.
Why this cannot be worked around in CI
I tried the obvious fixes first. None of them work:
You cannot filter the duplicate out in a workflow. I checked what GitHub records for each run, and every run created by the push — the real one and the duplicates, across all our workflows — points at the same push. Same event, same commit, same push. There is no field that tells the duplicate apart from the original, so there is nothing to write a condition against.
Concurrency settings do not help. A concurrency group can cancel a running job or make it wait, but it cannot stop the duplicate run from being created — and the checks appear as soon as the run is created. Turning off cancel-in-progress is worse: it doubles the CI bill, and a third event still cancels the queued second one.
Deleting the runs is the only cleanup GitHub offers, and deleting CI history to tidy up a display problem is not something we are willing to do.
Why it matters
- Reviewers misread healthy PRs as broken. One of our PRs showed 59 passing checks, 36 cancelled, 8 skipped, and 2 failed. Both failures were old entries already replaced by passing ones — the PR was genuinely green. But you cannot see that from the Checks tab. On a tall stack this happens to every layer, every time we restack.
- It wastes a lot of CI. Each duplicate run starts its full set of jobs (about 40 for us) before being cancelled. On a 17-PR stack that is several hundred wasted job starts per restack, on a shared runner pool.
What I measured
This is a 17-PR stack in a private repository, restacked several times in one day.
Before — one atomic push of the whole stack:
Out of 8 branches I sampled, 6 got duplicate CI runs.
After — same stack, same branches, pushed one at a time with a pause between each.
Each pause is its own experiment, because what matters for a given PR is the delay between pushing the branch below it and pushing the branch itself. I used pauses from 2 to 45 seconds. The times below are the real measured intervals, which include the time the push itself took:
| Pause between pushes | Branches | Duplicate runs |
|---|---|---|
| 3 seconds | 1 | 0 |
| 6–7 seconds | 2 | 0 |
| 11–12 seconds | 2 | 0 |
| 16–17 seconds | 2 | 0 |
| 21–22 seconds | 2 | 0 |
| 31 seconds | 2 | 0 |
| 46–47 seconds | 2 | 0 |
Zero duplicates in 14 tries in a row, compared with roughly 3-in-4 branches duplicating when pushed all at once. If the old rate still applied, getting 14 clean branches by luck would be about a 1-in-250-million shot.
Two honest caveats:
- I ran the shortest pauses first. If there were some warm-up effect, it would have made the short pauses look bad — they were clean, so that does not explain the result.
- This was one restack during a quiet period. I would not claim 3 seconds is a guaranteed floor. What I would claim is that a few seconds is plenty, and a minute is unnecessary.
What I am asking for
An opt-in flag that keeps today's behaviour by default:
gh stack sync --stagger 5s
gh stack submit --stagger 5s
gh stack push --stagger 5s
With the flag set, push the branches one at a time from the bottom up, pausing for the given duration between each, instead of pushing them all at once.
For a 15-branch stack at 5 seconds, that is a little over a minute — much cheaper than the CI the duplicate runs burn, and far cheaper than the confusion they cause reviewers.
The trade-off, stated plainly
Pausing between pushes means giving up the atomic push, so a failure partway through would leave the stack half pushed. That is already how gh stack push behaves today — its help text says "Updates are not atomic: a branch may update even if another branch is rejected" — so this would not introduce a new kind of failure. It is a good reason to keep the flag opt-in and to mention the trade-off in its help text.
Things I considered and ruled out
| Idea | Why it does not work |
|---|---|
| Filter out the duplicate event in the workflow | The duplicate is identical to the original, right down to the push it points at. Nothing to filter on |
| Change the concurrency settings | Cannot stop the run being created, and the checks appear with the run |
| Delete the cancelled runs | Works, but destroys CI history to fix a cosmetic problem |
| Document it and tell reviewers to check elsewhere | We already do. Reviewers look at the Checks tab, not at our contributing guide |
| Write our own staggered push script | This is what we are doing now. It works, but it means stepping outside the tool and losing its safety behaviour — which is why I would rather see it supported |
Environment
gh-stackv0.1.0- GitHub.com, self-hosted runners
- Stack size: 17 PRs
Activity
david-waltermire commentedon Aug 23, 2026
I have an implementation of this ready on a branch, but I cannot open the pull request myself:
POST /repos/github/gh-stack/pullsreturns 404 for a fork head, and none of the last 100 PRs on this repo originate from a fork. It looks like outside contributions are restricted, which is worth knowing given CONTRIBUTING.md invites forks. Happy to be told I have got that wrong.The branch is public, so a maintainer can open it directly:
main...david-waltermire:gh-stack:david-waltermire/stagger
Branch:
david-waltermire/staggerondavid-waltermire/gh-stack, one commit, based onmainatab00aa4.What it does
Adds
--stagger <duration>tosync,submit, andpush. With the flag absent or zero, behaviour is unchanged:syncstill does its single--force-with-lease --atomicmulti-ref push. With a positive duration, branches are pushed one at a time bottom-up with per-branch--force-with-lease, sleeping between pushes and never after the last one.The three commands have distinct push sites, so all three take the flag, but the multi-ref path routes through one new helper (
pushBranchesincmd/utils.go) rather than duplicating the logic.submitalready pushed per-branch in a loop interleaved with PR creation, so it only gained the sleep.Negative durations are rejected with
ErrInvalidArgs, and the help text states the trade-off: staggering necessarily gives up--atomic, so a mid-sequence failure leaves the stack partly pushed. That is already the documented behaviour ofgh stack push, which is why the flag is opt-in.Tests
Table-driven, in the existing testify and
git.SetOpsstyle, covering flag parsing including negative and malformed values, that zero keeps the single multi-ref push, that a positive value pushes per-branch in bottom-up order, and that no sleep occurs after the final push. Sleeping is faked through a package-levelsleepBetweenPushesvariable, mirroring the existinggit.SetOpsswap idiom, since the codebase has no clock abstraction and a real sleep cannot prove the absence of a trailing one.The three core tests were confirmed failing against unwired code before the commands were routed through the helper, so they do guard the behaviour rather than merely accompany it.
Diff is roughly 82 lines of production code and 429 lines of tests across
cmd/, plus--staggerrows and a short note indocs/src/content/docs/reference/cli.md.go build ./...,go vet ./...andgo test -race -count=1 ./...all pass. Two files,internal/modify/state.goandinternal/stack/lock_windows.go, already failgofmt -lon unmodifiedmain; I left them alone.Glad to revise the approach, the flag name, or the surface it applies to.
darist commentedon Sep 8, 2026
Independent reproduction on a newer version, plus one correction and one severity escalation.
Reproduction
7-PR stack in a private monorepo with ~24 PR-triggered workflows,
gh-stackv0.1.1 (this issue reports v0.1.0). A singlegh stack syncproduced exactly 2 runs per workflow for every mid-stack PR — uniformly 24 workflows x 2 runs, on 6 of the 7 PRs.I can confirm your bottom-of-stack observation. Runs-per-workflow tracked the number of ref updates affecting each PR exactly:
The PR timeline API is the one place the two distinct causes are visible:
base_ref_force_pushedandhead_ref_force_pushedarrive as separate events a couple of seconds apart (base at:13, head at:15in my sample), even thoughsyncissues a single atomic multi-ref push. GitHub appears to emit per-ref events regardless of how they were pushed. The ordering varies between PRs and doesn't change the outcome — only the count matters.The impact can be worse than cosmetic: PRs get hard-blocked
This issue frames the cancelled checks as reviewer confusion plus CI waste. In our case they blocked merges outright.
gh pr view --json mergeStateStatusreturnedBLOCKEDon the mid-stack PR I inspected in detail, with required contexts never reporting, because branch protection reads the latest check run per context and the latest wascancelled. Cancelled required contexts were present on all six affected PRs. Since GitHub doesn't guarantee which of two simultaneous runs gets cancelled, this isn't intermittent — it hit every mid-stack PR on every sync.Clearing it meant rerunning the cancelled runs per PR (
statusCheckRollup-> filterCANCELLED->gh run rerun), roughly 17-24 runs each, and it recurred on the next restack. Rebasing to force fresh runs isn't an option, since that drops existing approvals.Worth adding to the "why it matters" section, I think — it turns a display annoyance into a merge blocker on stacks with required checks.
Small correction to "there is no field that tells the duplicate apart"
The equivalent issue in git-spice (abhinav/git-spice#966) reports that the two
synchronizeevents are identical inbefore,after, and action type, but differ inpull_request.base.sha. That's consistent with the two-ref-update cause, and it means the events aren't strictly indistinguishable.This doesn't rescue either of your complaints — you're right that checks are posted at run creation, so nothing stops the duplicate run existing, and the waste is unaffected. But it does give a partial mitigation for the blocking symptom specifically. Adding
base.shato the concurrency group puts the two duplicates in different groups, so neither cancels the other and both reach a terminal state:Normal stale-run cancellation survives, because successive pushes to a head branch share a
base.sha. The cost is 2x CI on restacks, so it's strictly worse than--stagger, which prevents the duplicate altogether. Mentioning it only because it unblocks merges today without waiting on a tool change.On
--staggerI derived the same mitigation independently before finding this issue, and my timing data is consistent with your ~3 second floor: ref updates at
:13and:15produced runs created at:18-:19, so the duplicate window is just a few seconds of event-processing latency. +1 on the flag, and the atomicity trade-off looks clearly acceptable as opt-in.One more data point for the case: git-spice#966 shows the same duplicate-
synchronizepattern in an unrelated stacking tool, and is still open. That suggests this is platform behaviour rather than anything specific to gh-stack, and that a staggered push may be the only fix available at the tool level.Disclosure: this comment was researched and drafted with Claude Code, then reviewed by me before posting. The run counts, timings, and API observations above come from real workflow runs and PR timeline data in our repository — they are measured, not synthesized.
ci: expand pull_request branch filter to all branches for stacked PRs
fix(merge-status): count superseded cancelled runs against required c…
fix(checks): keep a required context's latest-run cancellation failing
fix(merge-status): report required checks that block stacked layers (#…