fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) by myasnikovdaniil · Pull Request #3146 · cozystack/cozystack

myasnikovdaniil · 2026-06-30T07:22:17Z

What

Backports the upstream Kamaji fix clastix/kamaji#1122 (which fixes clastix/kamaji#1115) as a source patch on top of the pinned 26.3.6-edge build. No version bump, no CRD/RBAC changes.

This lands the fix on main; the backport / backport-previous labels fan it out to release-1.5 and release-1.4 via the backport automation. Supersedes the per-release PRs #3021 (release-1.5) and #3020 (release-1.4).

Why

Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes — e.g. a tenant deleted before any Kubernetes app / TCP is ever created in it — leaves the DataStore stuck Terminating with finalizer kamaji.clastix.io/TenantControlPlane. That hangs helm uninstall <etcd> (which waits by default), freezes the etcd HelmRelease at status=uninstalling, and wedges the tenant namespace in Terminating indefinitely. Tracked in #3018.

Root cause: in 26.3.x-edge, DataStore deletion drives an AllowedDeletion condition that only transitions False → True. With 0 referencing TCPs at deletion time the condition is never initialized, so the reconcile falls through to logging DataStore can be safely deleted and the deferred finalizer-removal (gated on IsStatusConditionTrue) never runs — with no requeue to retry. The pinned 26.3.6-edge predates the upstream fix (merged 2026-04-14, first released in 26.4.4-edge).

Change

Adds packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff, applied by the existing RUN git apply /patches/*.diff step in the kamaji image build. The one-line guard adds the missing 0-TCP path:

- if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) {
+ if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) || len(tcpList.Items) == 0 {

Test

git apply --check confirms the patch applies cleanly to the 26.3.6-edge source pinned on main; after applying, the guard is present at controllers/datastore_controller.go.
Reproduced the deadlock on a v1.4.x cluster (tenant deleted before any TCP existed) and confirmed the manual unstick; this patch removes the need for it.

Workaround for an already-stuck DataStore (0 TCP refs): kubectl patch datastore <name> --type=merge -p '{"metadata":{"finalizers":null}}'.

Relates to #2885 (full kamaji bump for CVEs — supersedes this patch once it lands).

Fix tenant namespace stuck `Terminating` when an etcd DataStore with no referencing TenantControlPlanes is deleted (Kamaji datastore unused-deletion deadlock).

Summary by CodeRabbit

Bug Fixes
- Data stores are now recognized as unused even when no matching tenant control plane resources are found, improving deletion handling in that case.
- This prevents inactive data stores from being left behind when there are no associated matches.

gemini-code-assist · 2026-06-30T07:22:23Z

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

coderabbitai · 2026-06-30T07:22:34Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2bc64c76-645f-4059-a3c1-e8ced4f2d472

📥 Commits

Reviewing files that changed from the base of the PR and between 7722fef and 47638f8.

📒 Files selected for processing (1)

packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff

🚧 Files skipped from review as they are similar to previous changes (1)

packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff

📝 Walkthrough

Walkthrough

A patch file updates the Kamaji datastore controller logic so the "DataStore is not used" deletion path also applies when no TenantControlPlane objects are present.

Changes

Kamaji DataStore unused-deletion patch

Layer / File(s)	Summary
Extend DataStore.Reconcile unused-deletion condition `packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff`	Adds `len(tcpList.Items) == 0` alongside the existing `DataStoreConditionAllowedDeletionType` false check for the reconcile branch.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related issues

Tenant deletion wedges namespace in Terminating: pinned kamaji 26.3.5-edge predates upstream DataStore-finalizer fix (clastix/kamaji#1115/#1122) #3018: Describes the same DataStore deletion behavior when there are zero TenantControlPlane objects.

Suggested reviewers: kvaps, lllamnyp, androndo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the backported datastore deletion deadlock fix described in the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix-kamaji-datastore-deletion-main

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…/kamaji#1122) Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Aleksei Sviridkin (lexfrei)

LGTM — one-line upstream backport (clastix/kamaji#1122) that fixes the DataStore unused-deletion deadlock; verified it applies cleanly to the pinned 26.3.6-edge source and matches the merged upstream fix verbatim.

Business context: deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes wedges the DataStore in Terminating (finalizer never removed), hanging tenant namespace deletion — this backports the upstream guard.

Non-blocking follow-ups

When kamaji is bumped to >= 26.4.4-edge (the version that already contains this fix), remove packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff — otherwise the git apply /patches/*.diff build step fails, since the context line will already include the guard.

github-actions · 2026-07-03T09:52:34Z

Successfully created backport PR for release-1.5:

[Backport release-1.5] fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3187

github-actions · 2026-07-03T09:52:34Z

Successfully created backport PR for release-1.4:

[Backport release-1.4] fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3188

…n deadlock fix (clastix/kamaji#1122) (#3188) # Description Backport of #3146 to `release-1.4`.

…n deadlock fix (clastix/kamaji#1122) (#3187) # Description Backport of #3146 to `release-1.5`.

myasnikovdaniil requested review from IvanHunters, Andrey Kolkov (androndo), Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp) and Kirill Ilin (sircthulhu) as code owners June 30, 2026 07:22

github-actions Bot added size/S This PR changes 10-29 lines, ignoring generated files area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug labels Jun 30, 2026

dosubot Bot added backport Should change be backported on previous release backport-previous Backport target — previous release line labels Jun 30, 2026

myasnikovdaniil added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) and removed area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review labels Jun 30, 2026

This was referenced Jun 30, 2026

fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3021

Closed

fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3020

Closed

github-actions Bot added the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jun 30, 2026

Aleksei Sviridkin (lexfrei) added area/kamaji Issues or PRs related to Kamaji (hosted control planes for tenant Kubernetes) and removed area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review labels Jul 1, 2026

myasnikovdaniil force-pushed the fix-kamaji-datastore-deletion-main branch from 7722fef to 47638f8 Compare July 3, 2026 08:05

github-actions Bot added the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jul 3, 2026

Aleksei Sviridkin (lexfrei) approved these changes Jul 3, 2026

View reviewed changes

myasnikovdaniil merged commit 1e5f538 into main Jul 3, 2026
17 checks passed

myasnikovdaniil deleted the fix-kamaji-datastore-deletion-main branch July 3, 2026 09:52

This was referenced Jul 3, 2026

[Backport release-1.5] fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3187

Merged

[Backport release-1.4] fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) #3188

Merged

myasnikovdaniil added a commit that referenced this pull request Jul 3, 2026

[Backport release-1.4] fix(kamaji): backport datastore unused-deletio…

cf7dc6c

…n deadlock fix (clastix/kamaji#1122) (#3188) # Description Backport of #3146 to `release-1.4`.

myasnikovdaniil added a commit that referenced this pull request Jul 3, 2026

[Backport release-1.5] fix(kamaji): backport datastore unused-deletio…

ccb325e

…n deadlock fix (clastix/kamaji#1122) (#3187) # Description Backport of #3146 to `release-1.5`.

Navigation Menu

Uh oh!

fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) - #3146

fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)#3146
myasnikovdaniil merged 1 commit into
mainfrom
fix-kamaji-datastore-deletion-main

myasnikovdaniil commented Jun 30, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

myasnikovdaniil commented Jun 30, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Change

Test

Summary by CodeRabbit

Uh oh!

gemini-code-assist Bot commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Choose a reason for hiding this comment

Non-blocking follow-ups

Uh oh!

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

myasnikovdaniil commented Jun 30, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading