Skip to content

ci: guard helm index.yaml publish with a GCS generation precondition - #29793

Merged
f0ssel merged 5 commits into
mainfrom
f0ssel/helm-index-generation-lock
Sep 23, 2026
Merged

f0ssel merged 5 commits into
mainfrom
f0ssel/helm-index-generation-lock

Conversation

@f0ssel

@f0ssel f0ssel commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

The Publish Helm Chart step does an unguarded read-modify-write of gs://helm.coder.com/v2/index.yaml: download, helm repo index --merge with only this run's three charts, upload. The workflow concurrency group is keyed on github.ref, so releases for different tags run in parallel. When two of them reach this step at about the same time, both merge into the same old index and the second upload silently drops the first release's entries. This is how 2.36.6 (and, going by the current index, probably 2.36.2) went missing from the coder-v2 chart repo after back-to-back manual re-runs.

Fix

Treat the index like a compare-and-swap:

  • Read the object's GCS generation before and after downloading index.yaml, and fail if it changed underneath us.
  • Upload with x-goog-if-generation-match:<generation> so a concurrent writer causes a 412 instead of a clobber.
  • On either failure, exit with a ::error:: telling the operator to re-run the job. No automatic retry; a re-run merges against the fresh index.

Chart tarballs are still uploaded before the index, so an index entry is never published ahead of its tarball. The OCI pushes to ghcr.io/coder/chart are unchanged; they have no shared index and were never affected.

Serializing all release runs through a single concurrency group was considered and rejected: GitHub keeps only the newest pending run per group, so a third dispatch would silently cancel a queued release.

Verification

  • actionlint and shellcheck on the step script pass.
  • Exercised the step locally with a fake gsutil: the happy path uploads and bumps the generation; bumping the generation between read and upload makes the precondition fail and the step exit 1 with the re-run message.
  • Not exercised against the real bucket. The x-goog-if-generation-match header and gsutil stat Generation: field are standard GCS behaviour, but a dry-run cannot cover this because the step is skipped when dry_run is set.

This does not repair the already-missing entries; #29794 adds a one-off workflow for that.


Generated by Coder Agents on behalf of @f0ssel.

Concurrent release runs each read index.yaml, merge only their own
charts, and overwrite it, so the last writer drops the other release's
entries. This is what left 2.36.6 out of the coder-v2 chart repo.

Read the object's generation, upload with x-goog-if-generation-match,
and re-fetch and re-merge when the precondition fails.
@f0ssel

f0ssel commented Sep 23, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T15:51:11.344601Z 0ea968f Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 0ea968f0ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A precondition failure means another release published concurrently.
Surface it as a clear error and let the operator re-run the job.
Different tags previously ran in parallel because the concurrency group
included github.ref. Queue them instead so releases never publish to the
shared Helm index at the same time.
@f0ssel
f0ssel marked this pull request as ready for review September 23, 2026 16:17

@aslilac aslilac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense!

this seems like the kind of thing that might be nice to run in a backoff loop, even just retrying like once or twice would probably reduce the number of times you have to go in and click "retry" by hand quite a bit. but I'll leave that call up to you.

Comment thread .github/workflows/release.yaml Resolved
f0ssel pushed a commit that referenced this pull request Sep 23, 2026
Adds a manually dispatched `Helm index repair` workflow to re-add chart
entries that were dropped from `https://helm.coder.com/v2/index.yaml` by
the release race described in #29793. The 2.36.6 tarballs (and likely
2.36.2) are already in the bucket, on ghcr.io, and attached to the
GitHub release; only the index entry is missing, so the repair is a
merge and upload of `index.yaml`.

## Inputs

- `version`: chart version to re-add, for example `2.36.6` (a leading
`v` is stripped).
- `dry_run` (default `true`): do everything except the upload, and print
the exact `gsutil` command that would run.

## What it does

1. Copies whichever of `coder_helm_<v>.tgz`, `provisioner_helm_<v>.tgz`,
`ai-gateway_helm_<v>.tgz` exist in the bucket. Missing ones are warned
about (older versions predate the AI gateway chart); if none exist it
fails. Tarballs are taken from the bucket, never rebuilt, so the digests
written to the index match what is served.
2. Downloads `index.yaml`, pinned to a GCS generation (fails if the
object changes during the download, which means a release is
mid-publish).
3. Runs `helm repo index --merge`, then diffs the `chart version` list
before and after. Refuses to continue if any existing entry would be
removed. Exits cleanly if the entries are already present.
4. Prints the entries being added and a unified diff of `index.yaml`.
5. Dry run: prints the upload command and stops. Otherwise uploads with
`x-goog-if-generation-match` so it cannot clobber a concurrent write.

Gated behind the same maintainer/admin check as `release.yaml`, uses the
same GCP workload identity, `permissions: {}` at the top level with only
`id-token: write` on the repair job, and a workflow-level concurrency
group so two repairs cannot race each other.

## Verification

- `actionlint`, `shellcheck`, and `zizmor` pass.
- Ran the step script locally against a fake `gsutil` backed by a
directory: dry run leaves the bucket untouched and prints the command;
real run adds `coder 2.36.6` and `provisioner 2.36.6` while keeping the
existing entry and bumping the generation; re-running is a no-op; a
version with no tarballs fails.
- Not run against the real bucket. First real use should be a `dry_run:
true` dispatch for `2.36.6` to confirm the tarballs are found and the
diff looks right.

---
_Generated by Coder Agents on behalf of @f0ssel._
Co-authored-by: McKayla はな  <kayla@tree.camp>
@f0ssel
f0ssel merged commit 54bd34d into main Sep 23, 2026
27 of 28 checks passed
@f0ssel
f0ssel deleted the f0ssel/helm-index-generation-lock branch September 23, 2026 19:22
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants