Skip to content

ci: benchmark ARM64 runners for bottleneck test suites - #8203

Closed
nicohrubec wants to merge 2 commits into
masterfrom
ci/benchmark-arm-bottleneck-suites
Closed

nicohrubec wants to merge 2 commits into
masterfrom
ci/benchmark-arm-bottleneck-suites

Conversation

@nicohrubec

@nicohrubec nicohrubec commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Description

Move AI, AI Workflow, and Web 1 (28 matrix jobs) to ubuntu-22.04-arm to measure CI performance. Other jobs keep their existing runners. The initial one-line README wording change triggered the x86 baseline before the runner-only commit.

Issues

CI performance experiment; no linked issue.

Validation

Both first-attempt PR runs passed all 154 Test workflow jobs (152 matrix jobs, the required gate, and coverage reporting). Both used base commit d565fed3932d83a04731cecc0a7910a916888de3, the same SDK code, and the same configured test matrix. Workflow regeneration passes --fail-on-changes; mypy reports no issues in 203 source files.

Metric x86 baseline ARM change Difference
Required CI checks complete 11m31s 10m46s 45s faster (6.5%)
Full Test workflow, including coverage 11m59s 11m15s 44s faster (6.1%)
AI, all 10 jobs complete 11m26s 9m51s 1m35s faster (13.8%)
AI Workflow, all 6 jobs complete 10m39s 10m16s 23s faster (3.6%)
Web 1, all 12 jobs complete 10m27s 10m41s 14s slower (2.2%)

Suite completion times run from the Test workflow start until the last job in that suite finishes, including runner setup and queue delays. The critical path changed from AI/Python 3.13 to Web 1/Python 3.6.

Average individual job runtime increased: AI 7m03s to 7m15s (+2.8%), AI Workflow 7m51s to 8m55s (+13.4%), and Web 1 5m40s to 6m07s (+7.9%). Total job runtime across the 28 changed jobs increased 7.4%. This single paired run shows a shorter completion tail, but does not establish a consistent ARM speedup; unchanged x86 jobs also varied between runs.

Baseline run · ARM run

Co-Authored-By: GPT-6 <codex@openai.com>
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Results 📊

✅ 137221 passed | ⏭️ 7456 skipped | Total: 144677 | Pass Rate: 94.85% | Execution Time: 442m 22s

📊 Comparison with Base Branch

Metric Change
Total Tests —
Passed Tests —
Failed Tests —
Skipped Tests —

✨ Test counts unchanged from base.

All tests are passing successfully.

✅ Patch coverage is 100.00% (no changed executable lines found; target 80%).
Project statement coverage is 90.31%.


Generated by Coverage Action

Co-Authored-By: GPT-6 <codex@openai.com>
@nicohrubec nicohrubec closed this Nov 6, 2026
sentrivana added a commit that referenced this pull request Nov 12, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [astral-sh/setup-uv](https://redirect.github.com/astral-sh/setup-uv) |
| patch | `v10.3.0` → `v10.3.3` |
| [astral-sh/setup-uv](https://redirect.github.com/astral-sh/setup-uv) |
action | patch | `v10.3.0` → `v10.3.3` |
| [getsentry/craft](https://redirect.github.com/getsentry/craft) |
action | minor | `3.9.0` → `3.12.0` |
| [getsentry/craft](https://redirect.github.com/getsentry/craft) |
workflow | minor | `3.9.0` → `3.12.0` |
| [java-jdk](https://adoptium.net) | uses-with | major | `21` → `25` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/6715) for more information.

---

### Release Notes

<details>
<summary>astral-sh/setup-uv</summary>

###
[`v10.3.3`](https://redirect.github.com/astral-sh/setup-uv/releases/tag/v10.3.3):
🌈 Use Astral mirror for version manifests with GitHub fallback

[Compare
Source](https://redirect.github.com/astral-sh/setup-uv/compare/v10.3.2...v10.3.3)

#### Changes

This release contains something that should have been included a very
long time ago. Sorry for sleeping on that.

This action now retrieves the release manifest that contains the
available versions, their download links and checksums from the Astral
mirror, rather than trying GitHub first. This should present much more
stable.
(Known checksums are still checked, so the security impact does not
change)

Also a big thank you to
[@&#8203;soujasK](https://redirect.github.com/soujasK) for a set of
bugfixes 👏

#### 🐛 Bug fixes

- Use Astral mirror for version manifests with GitHub fallback
[@&#8203;eifinger](https://redirect.github.com/eifinger)
([#&#8203;1119](https://redirect.github.com/astral-sh/setup-uv/issues/1119))
- fix: allow whitespace in uv requirement specifiers
[@&#8203;soujasK](https://redirect.github.com/soujasK)
([#&#8203;1112](https://redirect.github.com/astral-sh/setup-uv/issues/1112))
- fix: resolve \~= specifiers with PEP 440 semantics
[@&#8203;soujasK](https://redirect.github.com/soujasK)
([#&#8203;1111](https://redirect.github.com/astral-sh/setup-uv/issues/1111))
- fix: only report a saved cache when saving succeeded
[@&#8203;soujasK](https://redirect.github.com/soujasK)
([#&#8203;1110](https://redirect.github.com/astral-sh/setup-uv/issues/1110))

#### 🧰 Maintenance

- chore: update known checksums for 0.13.6
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1117](https://redirect.github.com/astral-sh/setup-uv/issues/1117))

#### 📚 Documentation

- docs: use JSDoc for compatible release specifier comment
[@&#8203;eifinger](https://redirect.github.com/eifinger)
([#&#8203;1113](https://redirect.github.com/astral-sh/setup-uv/issues/1113))
- docs: update version references to v10.3.2
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1104](https://redirect.github.com/astral-sh/setup-uv/issues/1104))

###
[`v10.3.2`](https://redirect.github.com/astral-sh/setup-uv/releases/tag/v10.3.2):
🌈 update known checksums for 0.13.5

[Compare
Source](https://redirect.github.com/astral-sh/setup-uv/compare/v10.3.1...v10.3.2)

#### Changes

#### 🧰 Maintenance

- chore: update known checksums for 0.13.5
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1103](https://redirect.github.com/astral-sh/setup-uv/issues/1103))

#### 📚 Documentation

- docs: update version references to v10.3.1
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1102](https://redirect.github.com/astral-sh/setup-uv/issues/1102))

###
[`v10.3.1`](https://redirect.github.com/astral-sh/setup-uv/releases/tag/v10.3.1):
🌈 preserve cache error class names in save-cache bundle

[Compare
Source](https://redirect.github.com/astral-sh/setup-uv/compare/v10.3.0...v10.3.1)

##### Changes

This release includes a fix so that multiple concurrent workflows trying
to upload the same cache no longer warn

`##[warning]Failed to save: Unable to reserve cache with key ...,
another job may be creating this cache.`

##### 🐛 Bug fixes

- fix: preserve cache error class names in save-cache bundle
[@&#8203;dannyward630](https://redirect.github.com/dannyward630)
([#&#8203;1093](https://redirect.github.com/astral-sh/setup-uv/issues/1093))

##### 🧰 Maintenance

- chore: update known checksums for 0.13.4
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1099](https://redirect.github.com/astral-sh/setup-uv/issues/1099))
- chore: update known checksums for 0.13.2/0.13.3
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1098](https://redirect.github.com/astral-sh/setup-uv/issues/1098))
- chore: update known checksums for 0.13.1
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1094](https://redirect.github.com/astral-sh/setup-uv/issues/1094))
- Verify npm dependency signatures before builds
[@&#8203;zaniebot](https://redirect.github.com/zaniebot)
([#&#8203;1027](https://redirect.github.com/astral-sh/setup-uv/issues/1027))

##### 📚 Documentation

- docs: cache-dependency-glob matches relative to the working directory
[@&#8203;flange-ipb](https://redirect.github.com/flange-ipb)
([#&#8203;1092](https://redirect.github.com/astral-sh/setup-uv/issues/1092))
- docs: update version references to v10.3.0
@&#8203;[github-actions\[bot\]](https://redirect.github.com/apps/github-actions)
([#&#8203;1090](https://redirect.github.com/astral-sh/setup-uv/issues/1090))

##### ⬆️ Dependency updates

- chore(deps): roll up Dependabot updates
[@&#8203;eifinger](https://redirect.github.com/eifinger)
([#&#8203;1101](https://redirect.github.com/astral-sh/setup-uv/issues/1101))

</details>

<details>
<summary>getsentry/craft</summary>

###
[`v3.12.0`](https://redirect.github.com/getsentry/craft/blob/HEAD/CHANGELOG.md#3120)

[Compare
Source](https://redirect.github.com/getsentry/craft/compare/3.11.2...3.12.0)

NuGet publishing now supports GitHub OIDC trusted publishing while
retaining token-based fallback, giving developers a configurable
alternative for package authentication.

##### New Features ✨

NuGet publishing can use GitHub OIDC trusted publishing, with
configurable identity and unchanged token-based fallback.

- (nuget) Support nuget.org trusted publishing with GitHub OIDC by
[@&#8203;ric-oliv](https://redirect.github.com/ric-oliv) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;972](https://redirect.github.com/getsentry/craft/pull/972)

###
[`v3.11.2`](https://redirect.github.com/getsentry/craft/blob/HEAD/CHANGELOG.md#3112)

[Compare
Source](https://redirect.github.com/getsentry/craft/compare/3.11.1...3.11.2)

Changelog credits retain co-author attribution after squash merges,
while batched GraphQL lookups reduce requests and tolerate partial
results.

##### Bug Fixes 🐛

Changelog crediting now preserves co-author attribution after squash
merges, while batching GraphQL lookups reduces requests and accepts
partial results.

- (changelog) Credit PR commit co-authors when the merge commit has none
by [@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;970](https://redirect.github.com/getsentry/craft/pull/970)

###
[`v3.11.1`](https://redirect.github.com/getsentry/craft/blob/HEAD/CHANGELOG.md#3111)

[Compare
Source](https://redirect.github.com/getsentry/craft/compare/3.11.0...3.11.1)

Floating GitHub tags now resolve to valid stable releases across
preparation, versioning, and changelog operations.

##### Bug Fixes 🐛

GitHub version lookups select valid stable releases for floating tags
across prepare, versioning, and changelog operations.

- (github) Skip prereleases and ignore floating tags in version lookup
by [@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;969](https://redirect.github.com/getsentry/craft/pull/969)

###
[`v3.11.0`](https://redirect.github.com/getsentry/craft/blob/HEAD/CHANGELOG.md#3110)

[Compare
Source](https://redirect.github.com/getsentry/craft/compare/3.10.0...3.11.0)

Larger changelogs gain bounded AI summarization, while publishing
surfaces merge failures, handles narrow version conflicts, and waits for
npm availability.

##### New Features ✨

Configurable AI summary budgets and clearer timeout diagnostics
accommodate larger changelogs without changing request limits.

##### Changelog

AI summary timing is now configurable through aiSummary.timeout. The
default total budget remains 120 seconds; integer values from 30 to 1800
are accepted. Set aiSummary.timeout to 0 to remove the total budget,
while the 30-second per-request limit and 64-request cap still apply.
Warnings now distinguish request-timeout from total-budget deadline
failures.

- Allow disabling the AI summary total timeout by
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;968](https://redirect.github.com/getsentry/craft/pull/968)
- Make the AI summary timeout configurable by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;967](https://redirect.github.com/getsentry/craft/pull/967)

##### Bug Fixes 🐛

Publishing exposes merge failures, safely retries completed targets,
resolves limited version conflicts, and waits for npm availability.

##### Publish

Publishing now fails when merging the release branch back fails, instead
of reporting success after targets ship. The retained publish state
allows retries to skip completed targets. Version-bump-only conflicts
can be resolved automatically when every conflicted file matches the
supported version-token pattern; other conflicts continue to the
existing conflict handling.

- Fail the publish when the release branch merge fails by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;965](https://redirect.github.com/getsentry/craft/pull/965)
- Resolve merge conflicts caused only by the version bump by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;964](https://redirect.github.com/getsentry/craft/pull/964)

##### Other

- (targets) Wait for npm before publishing to the MCP Registry by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;963](https://redirect.github.com/getsentry/craft/pull/963)

###
[`v3.10.0`](https://redirect.github.com/getsentry/craft/blob/HEAD/CHANGELOG.md#3100)

[Compare
Source](https://redirect.github.com/getsentry/craft/compare/3.9.0...3.10.0)

Deployments now wait for builds and expose failures, while release
tooling supports Craft 3 and prevents accidental pre-1.0 major version
jumps.

##### New Features ✨

GitHub Pages deployments now wait for builds and report failures with
deployment details.

- (gh-pages) Wait for the Pages build and retry when it fails by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;961](https://redirect.github.com/getsentry/craft/pull/961)

##### Bug Fixes 🐛

The reusable release workflow now uses the Craft v3 action, enabling
Craft 3 configuration and current release-blocker handling. For versions
below 1.0.0, craft prepare auto and craft prepare pre convert
commit-selected major bumps to minor versions. Use craft prepare major
or an explicit exact version to release 1.0.0. JSON changelog output and
PR previews still show the raw major bump type.

- (release) Use craft v3 action in reusable release workflow by
[@&#8203;BYK](https://redirect.github.com/BYK) with
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;959](https://redirect.github.com/getsentry/craft/pull/959)
- (version) Keep auto and pre below 1.0.0 on breaking changes by
[@&#8203;sentry-junior](https://redirect.github.com/sentry-junior) in
[#&#8203;962](https://redirect.github.com/getsentry/craft/pull/962)

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/getsentry/sentry-python).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNjQuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIiwibGFiZWxzIjpbXX0=-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Ivana Kellyer <131587164+sentrivana@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant