Repository navigation
ci: guard helm index.yaml publish with a GCS generation precondition - #29793
Conversation
Concurrent release runs each read index.yaml, merge only their own charts, and overwrite it, so the last writer drops the other release's entries. This is what left 2.36.6 out of the coder-v2 chart repo. Read the object's generation, upload with x-goog-if-generation-match, and re-fetch and re-merge when the precondition fails.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
A precondition failure means another release published concurrently. Surface it as a clear error and let the operator re-run the job.
Different tags previously ran in parallel because the concurrency group included github.ref. Queue them instead so releases never publish to the shared Helm index at the same time.
This reverts commit 198f5c7.
aslilac
left a comment
There was a problem hiding this comment.
makes sense!
this seems like the kind of thing that might be nice to run in a backoff loop, even just retrying like once or twice would probably reduce the number of times you have to go in and click "retry" by hand quite a bit. but I'll leave that call up to you.
Adds a manually dispatched `Helm index repair` workflow to re-add chart entries that were dropped from `https://helm.coder.com/v2/index.yaml` by the release race described in #29793. The 2.36.6 tarballs (and likely 2.36.2) are already in the bucket, on ghcr.io, and attached to the GitHub release; only the index entry is missing, so the repair is a merge and upload of `index.yaml`. ## Inputs - `version`: chart version to re-add, for example `2.36.6` (a leading `v` is stripped). - `dry_run` (default `true`): do everything except the upload, and print the exact `gsutil` command that would run. ## What it does 1. Copies whichever of `coder_helm_<v>.tgz`, `provisioner_helm_<v>.tgz`, `ai-gateway_helm_<v>.tgz` exist in the bucket. Missing ones are warned about (older versions predate the AI gateway chart); if none exist it fails. Tarballs are taken from the bucket, never rebuilt, so the digests written to the index match what is served. 2. Downloads `index.yaml`, pinned to a GCS generation (fails if the object changes during the download, which means a release is mid-publish). 3. Runs `helm repo index --merge`, then diffs the `chart version` list before and after. Refuses to continue if any existing entry would be removed. Exits cleanly if the entries are already present. 4. Prints the entries being added and a unified diff of `index.yaml`. 5. Dry run: prints the upload command and stops. Otherwise uploads with `x-goog-if-generation-match` so it cannot clobber a concurrent write. Gated behind the same maintainer/admin check as `release.yaml`, uses the same GCP workload identity, `permissions: {}` at the top level with only `id-token: write` on the repair job, and a workflow-level concurrency group so two repairs cannot race each other. ## Verification - `actionlint`, `shellcheck`, and `zizmor` pass. - Ran the step script locally against a fake `gsutil` backed by a directory: dry run leaves the bucket untouched and prints the command; real run adds `coder 2.36.6` and `provisioner 2.36.6` while keeping the existing entry and bumping the generation; re-running is a no-op; a version with no tarballs fails. - Not run against the real bucket. First real use should be a `dry_run: true` dispatch for `2.36.6` to confirm the tarballs are found and the diff looks right. --- _Generated by Coder Agents on behalf of @f0ssel._
Co-authored-by: McKayla はな <kayla@tree.camp>
The
Publish Helm Chartstep does an unguarded read-modify-write ofgs://helm.coder.com/v2/index.yaml: download,helm repo index --mergewith only this run's three charts, upload. The workflowconcurrencygroup is keyed ongithub.ref, so releases for different tags run in parallel. When two of them reach this step at about the same time, both merge into the same old index and the second upload silently drops the first release's entries. This is how 2.36.6 (and, going by the current index, probably 2.36.2) went missing from thecoder-v2chart repo after back-to-back manual re-runs.Fix
Treat the index like a compare-and-swap:
generationbefore and after downloadingindex.yaml, and fail if it changed underneath us.x-goog-if-generation-match:<generation>so a concurrent writer causes a412instead of a clobber.::error::telling the operator to re-run the job. No automatic retry; a re-run merges against the fresh index.Chart tarballs are still uploaded before the index, so an index entry is never published ahead of its tarball. The OCI pushes to
ghcr.io/coder/chartare unchanged; they have no shared index and were never affected.Serializing all release runs through a single
concurrencygroup was considered and rejected: GitHub keeps only the newest pending run per group, so a third dispatch would silently cancel a queued release.Verification
actionlintandshellcheckon the step script pass.gsutil: the happy path uploads and bumps the generation; bumping the generation between read and upload makes the precondition fail and the step exit 1 with the re-run message.x-goog-if-generation-matchheader andgsutil statGeneration:field are standard GCS behaviour, but a dry-run cannot cover this because the step is skipped whendry_runis set.This does not repair the already-missing entries; #29794 adds a one-off workflow for that.
Generated by Coder Agents on behalf of @f0ssel.