CLDSRV-961 Tests for lifecycle listings over PHD master keys - #6250
Open
nicolas2bert wants to merge 2 commits into
Open
CLDSRV-961 Tests for lifecycle listings over PHD master keys#6250nicolas2bert wants to merge 2 commits into
nicolas2bert wants to merge 2 commits into
Conversation
Contributor
Hello nicolas2bert,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
nicolas2bert
force-pushed
the
bugfix/CLDSRV-961/lc-phd
branch
from
August 13, 2026 10:18
42ec3b9 to
ca7fa09
Compare
❌ 2 Tests Failed:
View the full list of 3 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
bucketd runs the lifecycle listing, not cloudserver. The s3c legs therefore need the fix in the MetaData image, not only in the pinned arsenal. This points at a build of MetaData bugfix/MD-1359/lc-phd. Repin on a released tag when MD-1359 ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Lifecycle listings can get stuck forever on PHD master keys.
A PHD (placeholder) master is written when the current version of a key is deleted. A repair job later promotes the newest surviving version back to master. Until the repair runs, the master key holds a PHD value. If the repair never runs, or if no version survives, the PHD stays.
DelimiterVersions accepted a PHD master and skipped it without touching the resume marker. That is fine for a plain version listing, which has no scan budget. It is not fine for the two lifecycle listings, DelimiterNonCurrent and DelimiterOrphanDeleteMarker. Both count every scanned entry against maxScannedLifecycleListingEntries and truncate when the budget runs out.
So a run of PHD masters longer than the scan limit burns the whole budget without moving the marker. The listing returns IsTruncated: true with no NextKeyMarker. The next listing starts at the same place and returns the same thing. Lifecycle never gets past that point, and nothing beyond it is ever expired.
Example
Keyspace (v0), scan limit 3:
phd-1 .. phd-6 dangling PHD masters
zebra\0v1 current
zebra\0v2 noncurrent -> must be expired
Before:
page 1: scans phd-1, phd-2, phd-3 -> truncated, no NextKeyMarker
page 2: identical request -> identical result
...forever. zebra\0v2 is never expired.
After:
page 1: phd-1, phd-2, phd-3 -> truncated, NextKeyMarker=phd-2
page 2: phd-3, phd-4, phd-5 -> truncated, NextKeyMarker=phd-4
page 3: phd-5, phd-6, zebra\0v1, zebra\0v2 -> zebra\0v2 listed
What changed
DelimiterVersions — new handlePHDMaster hook
Both PHD branches (keyHandler_NotSkippingV0 and keyHandler_NotSkippingV1) now call handlePHDMaster(key, versionId, value) instead of returning FILTER_ACCEPT directly.
The default implementation returns FILTER_ACCEPT and changes no state. Plain version listings behave exactly as before.
DelimiterOrphanDeleteMarker — treat a PHD as a key transition
The override runs the same new-key branch as addVersion():
The second point matters. A delete marker is held in memory until an entry of another key proves it is an orphan. A PHD master is such an entry. If the marker moved past the candidate without emitting it, the candidate would never be scanned again, and the orphan delete marker would never expire.
scan limit 3, keyspace: banana\0v1 (delete marker), phd-1, phd-2, ...
banana\0v1 -> held as candidate
phd-1 -> new key: banana is proven orphan -> emitted, then marker moves
DelimiterNonCurrent — keep the marker one PHD key behind
The override sets nextKeyMarker to the previous PHD key, never to the key being scanned, and clears nextVersionIdMarker. The marker only moves forward.
A marker on the key being scanned would give the next listing a key-marker with no version-id-marker. S3 reads that as "start after every version of this key". The versions of a PHD master that still has some would be skipped, and they would never be expired. One key behind costs one re-scanned entry per truncation, and skips nothing.
The override leaves prevKey and staleDate untouched. This is deliberate. The next version key scanned is the newest surviving version under the PHD — the one the repair promotes back to master. It must stay classified as current:
apple (PHD) -> marker left behind apple, prevKey untouched
apple\0v1 -> first version seen for apple -> current, protected
apple\0v2 -> noncurrent -> expirable, staleDate = v1's date
Setting prevKey = 'apple' would make apple\0v1 look noncurrent. Noncurrent version expiration would then delete the very version the repair needs to promote. That is data loss.
One fallback remains: if the scan limit lands on the first PHD of the listing, there is no previous PHD and no marker yet. The marker then points at that key, and that one key loses its versions for this pass. Progress matters more than one missed pass, and the alternative is the infinite loop described above.
Tests
All cases run against both v0 and v1 bucket formats.