[Backport release-1.6] fix(ci): overlay images from the PR base branch, and publish per-line artifacts - #3514
Conversation
The PR finalize job pulls cozystack-packages:main and repoints every package the PR did not rebuild at current-main images. On a main-based PR that is the whole point: it stops e2e from testing last-release images for everything outside the build matrix. On a release-line PR it is wrong — the committed refs there are not stale, they are that line`s released digests, and the charts are written against exactly those. So a release-line PR was installing main`s binaries onto its own charts, and the mismatch grows with every commit main gains. #3437 is the demonstration: a one-line change on release-1.6 that deactivates an app failed install deterministically, twice, with SchemaError(...core/v1alpha1.Option.spec): unknown model in reference: "...core~1v1alpha1.OptionSpec" from main`s cozystack-controller serving an aggregated OpenAPI that branch`s charts cannot validate against. Nothing in the PR was broken; the lane was. Left alone this makes every 1.6 backport look red, which is when release-branch PRs are busiest. Both overlay steps are now gated on `github.base_ref == main`, so a release-line PR keeps its committed digests — the behaviour that predates the overlay. Retargeting the overlay at a per-line artifact would be better but is not possible today: build-main.yaml publishes only cozystack-packages:main, and the registry carries no release-* equivalent (verified against the packages repo`s tag list: `main` plus per-PR tags, nothing else). hack/overlay-main-images_test.bats pins the wiring per step, so adding a third overlay step without the guard fails the suite. Mutation-checked by removing one guard. 12/12 green; actionlint clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> (cherry picked from commit fea40d6)
… artifacts Supersedes the interim guard in this branch`s first commit, which skipped the overlay entirely for release-line PRs. Skipping fixed the wrong images but left those PRs testing their line`s last release: for any package the PR did not rebuild, the committed ref is the released digest, so a component changed by an earlier backport was exercised as its pre-backport binary until the next rc. The overlay now reads `cozystack-packages:<base branch>` instead of always `:main`, and build-release.yaml publishes that artifact for every maintained `release-<major>.<minor>` branch the way build-main.yaml does for main: images tagged with the branch, and the whole packages tree pushed with each reference digest-pinned to what the run just built. Each base branch therefore has its own generation to overlay from, which is what the original bug was really about — #3437 failed install because main`s cozystack-controller served an aggregated OpenAPI release-1.6`s charts could not validate against. Three deliberate choices: * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The per-release and rc staging branches promote-rc.yaml and tags.yaml create (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their images come from the tag build, and rebuilding them would be waste. * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per image and build-main.yaml is deliberately its only, serialized writer so concurrent builds cannot race on the cache manifest (the 409 class #2711 fixed for image tags). A line build can overlap a main build, so writing here would reintroduce that race. Line builds read the cache. * A missing artifact still degrades to committed refs, but on a release line it now emits a ::warning:: naming the branch. Silent degradation is indistinguishable from a working overlay, which is how a mis-specified branch filter would hide for a whole release cycle. Cost: one `make build` per push to a maintained line, i.e. per merged backport. hack/overlay-main-images_test.bats pins the artifact tag to the base branch, rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main, setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test. 13/13 green; actionlint and zizmor clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> (cherry picked from commit 2330d6f)
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
## Why `release-1.6` was created at v1.6.0's merge commit on 2026-07-22, and the finalize fixes landed on `main` after that. Workflow files run from the ref they fire on, so a promote PR based on `release-1.6` runs **this** copy of `pull-requests-release.yaml`, not `main`'s. Shipping v1.6.1 from this line today would repeat two defects v1.6.0 already hit. ## What this carries Three upstream commits, after which this file is **byte-identical to `main`'s copy** — so later backports touching it will not conflict. | Upstream | Change | | --- | --- | | `01e1e7188` | `persist-credentials: false` on the checkout | | `f49d54a68` | publish the release with the merged changelog as its body | | `ba67fea7d` | drop `paths-ignore` from the trigger | ### persist-credentials Without it the checkout persists `GITHUB_TOKEN` as an `http.extraheader`, which silently wins over the app token injected by `git remote set-url`. The stable tag then pushes as `GITHUB_TOKEN`, which creates no workflow run, so `tags.yaml` never fires and its `generate-changelog` and `update-website-docs` backstops stay silent. That is exactly what happened to v1.6.0. ### Changelog as release body Without it the release publishes with the draft's own body, `"Promoted from vX.Y.Z-rc.N"`. **These two compound into a permanent defect on a maintenance line.** A patch's changelog is committed to `release-1.6.1`, merged into `release-1.6`, and never reaches `main` — and `update-releasenotes.yaml` only watches `main`, so nothing would ever sync it. The tag-time backstop that would have ported it is dead because of the credential bug. v1.6.1 would ship with placeholder release notes permanently, with no error anywhere. ### paths-ignore Latent rather than active. A promote PR carrying only `docs/changelogs/vX.Y.Z.md` would be dropped by the filter, producing no finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites so it would not have fired, but the filter has no remaining purpose now that the promote PR always carries a changelog. ## On the cherry-pick policy `docs/release.md`'s skip rule says CI-only changes do not belong in a patch. That rule governs release *contents*; backporting release *machinery* so the line is releasable at all is a different thing, and there is precedent already on this branch — #3473, #3474 and #3514 are all CI/release backports. ## Verification `actionlint` clean. `zizmor` clean, no findings. YAML parses. No bats suite references this workflow, and `hack/promote-gate-contract.bats` does not exist on this line, so there is no stale contract test to trip. The diff against `origin/main`'s copy of the file is empty. ## Not included Three known gaps on this line are deliberately out of scope, none of which needs a code change to ship v1.6.1: - `e2e-tag.yaml` is absent and `tags.yaml` has no `rc-e2e` job, so no E2E runs on the rc. Handled by dispatching **E2E Release Tag from `main`** against the published rc tag, which also supplies the promote gate's alternate evidence. - `pull-requests.yaml` has no `labeled` trigger, so adding `full-e2e` to the promote PR starts nothing. Backporting that needs the label-event guards too, and rc-time validation makes it unnecessary here. - finalize is still the pre-#3456 monolith, so a mid-registry failure is not re-runnable and needs the documented hand recovery. ```release-note NONE ```
What this PR does
Hand backport of #3471 to
release-1.6. Two clean cherry-picks, each carrying its-xreference; the resulting tree is byte-identical to #3471's merged state for all three files.The overlay must be fixed on this branch to have any effect here. For
pull_requestevents GitHub builds the workflow from the merge ref, and forpushevents it reads the workflow from the pushed ref — so neither half of #3471 reaches this line while it lives only onmain. Concretely,build-release.yamlonmainnever fires for a push torelease-1.6, and arelease-1.6PR keeps using this branch's copy ofpull-requests.yaml.Why the bot could not do this
The automatic backport reported success on every job and opened nothing (run 30817682877). It is not a conflict — #3471 carries an empty commit,
007d0b1a(chore(ci): re-trigger CI after a label event produced a no-op run, zero files).korthout/backport-actioncherry-picks commit-by-commit without--allow-empty, so that commit fails withThe previous cherry-pick is now empty; thedraft_commit_conflictsfallback then runsgit commit --all -m BACKPORT-CONFLICT, which also has nothing to commit, and the action aborts:This is a general gap, not specific to this PR: any labelled PR containing an empty commit backports to nothing, and the failure surfaces as three green jobs. Worth a follow-up on the workflow — skipping empty commits, or having the action's failure fail the job — but that is out of scope here.
What this unblocks
#3437 fails install deterministically on this line today, because the unguarded overlay hands it main's
cozystack-controllerto run againstrelease-1.6's charts. The binary indexesinternal.cozystack.io/v1alpha1 TenantProjectionat startup, this branch ships no such CRD, so it fatals and every release that depends on it stalls. With this merged, the overlay readscozystack-packages:release-1.6instead.That works even before the first
build-release.yamlrun publishes the artifact: a missing artifact degrades to the committed refs, which on this branch are the v1.6.0 digests its charts are written against. The degraded path is also no longer silent — it emits a warning naming the branch.#3437 will need a new head commit rather than a re-run, since the finalize job is reused on re-run and its existing
pr.patchalready has main's refs baked in.Note on cost
Merging this is itself a push to
release-1.6, sobuild-release.yamlfires on it and runs a fullmake build(up to 2h on a 24-cpu runner) to publish the line's images andcozystack-packages:release-1.6. That recurs per push to this line;concurrencywithcancel-in-progresscollapses bursts, andpaths-ignoreonly exemptsdocs/**.Verification
hack/overlay-main-images_test.bats13/13 green on this branch, including the two wiring tests the backport brings (the overlay reads the artifact for the PR's own base branch,every maintained release line publishes its own packages artifact).actionlintclean on both workflows, and both parse as YAML.hack/common-envs.mkis byte-identical betweenmainandrelease-1.6, soIMAGE_TAG,WRITE_CACHE,PUBLISH_VERSIONEDandPUBLISH_FLOATINGbehave here exactly as they do on main.Release note