Skip to content

CLDSRV-961 Tests for lifecycle listings over PHD master keys - #6250

Open
nicolas2bert wants to merge 2 commits into
development/9.4from
bugfix/CLDSRV-961/lc-phd
Open

CLDSRV-961 Tests for lifecycle listings over PHD master keys#6250
nicolas2bert wants to merge 2 commits into
development/9.4from
bugfix/CLDSRV-961/lc-phd

Conversation

@nicolas2bert

Copy link
Copy Markdown
Contributor

Problem

Lifecycle listings can get stuck forever on PHD master keys.

A PHD (placeholder) master is written when the current version of a key is deleted. A repair job later promotes the newest surviving version back to master. Until the repair runs, the master key holds a PHD value. If the repair never runs, or if no version survives, the PHD stays.

DelimiterVersions accepted a PHD master and skipped it without touching the resume marker. That is fine for a plain version listing, which has no scan budget. It is not fine for the two lifecycle listings, DelimiterNonCurrent and DelimiterOrphanDeleteMarker. Both count every scanned entry against maxScannedLifecycleListingEntries and truncate when the budget runs out.

So a run of PHD masters longer than the scan limit burns the whole budget without moving the marker. The listing returns IsTruncated: true with no NextKeyMarker. The next listing starts at the same place and returns the same thing. Lifecycle never gets past that point, and nothing beyond it is ever expired.

Example

Keyspace (v0), scan limit 3:

phd-1 .. phd-6 dangling PHD masters
zebra\0v1 current
zebra\0v2 noncurrent -> must be expired

Before:

page 1: scans phd-1, phd-2, phd-3 -> truncated, no NextKeyMarker
page 2: identical request -> identical result
...forever. zebra\0v2 is never expired.

After:

page 1: phd-1, phd-2, phd-3 -> truncated, NextKeyMarker=phd-2
page 2: phd-3, phd-4, phd-5 -> truncated, NextKeyMarker=phd-4
page 3: phd-5, phd-6, zebra\0v1, zebra\0v2 -> zebra\0v2 listed

What changed

DelimiterVersions — new handlePHDMaster hook

Both PHD branches (keyHandler_NotSkippingV0 and keyHandler_NotSkippingV1) now call handlePHDMaster(key, versionId, value) instead of returning FILTER_ACCEPT directly.

The default implementation returns FILTER_ACCEPT and changes no state. Plain version listings behave exactly as before.

DelimiterOrphanDeleteMarker — treat a PHD as a key transition

The override runs the same new-key branch as addVersion():

  • It advances prevKeyName / keyName, so truncation has a resume position.
  • It emits the held delete-marker candidate first.
  • It sets value = null, so a PHD is never listed as an orphan delete marker.

The second point matters. A delete marker is held in memory until an entry of another key proves it is an orphan. A PHD master is such an entry. If the marker moved past the candidate without emitting it, the candidate would never be scanned again, and the orphan delete marker would never expire.

scan limit 3, keyspace: banana\0v1 (delete marker), phd-1, phd-2, ...
banana\0v1 -> held as candidate
phd-1 -> new key: banana is proven orphan -> emitted, then marker moves

DelimiterNonCurrent — keep the marker one PHD key behind

The override sets nextKeyMarker to the previous PHD key, never to the key being scanned, and clears nextVersionIdMarker. The marker only moves forward.

A marker on the key being scanned would give the next listing a key-marker with no version-id-marker. S3 reads that as "start after every version of this key". The versions of a PHD master that still has some would be skipped, and they would never be expired. One key behind costs one re-scanned entry per truncation, and skips nothing.

The override leaves prevKey and staleDate untouched. This is deliberate. The next version key scanned is the newest surviving version under the PHD — the one the repair promotes back to master. It must stay classified as current:

apple (PHD) -> marker left behind apple, prevKey untouched
apple\0v1 -> first version seen for apple -> current, protected
apple\0v2 -> noncurrent -> expirable, staleDate = v1's date

Setting prevKey = 'apple' would make apple\0v1 look noncurrent. Noncurrent version expiration would then delete the very version the repair needs to promote. That is data loss.

One fallback remains: if the scan limit lands on the first PHD of the listing, there is no previous PHD and no marker yet. The marker then points at that key, and that one key loses its versions for this pass. Progress matters more than one missed pass, and the alternative is the infinite loop described above.

Tests

  • delimiterVersions.spec.js — guard: a PHD master is still not listed and still does not advance the marker in a plain version listing.
  • delimiterNonCurrent.spec.js — marker set inside a PHD run; the newest surviving version under a PHD stays protected; crawl across a PHD run longer than the scan limit with marker feedback; scan limit landing exactly on a PHD.
  • delimiterOrphanDeleteMarker.spec.js — NextMarker set inside a PHD run; a dangling PHD is never listed as an orphan; crawl across a PHD run; a held candidate is emitted when the PHD run begins.

All cases run against both v0 and v1 bucket formats.

@bert-e

bert-e commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-961 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.2

Please check the Fix Version/s of CLDSRV-961, or the target
branch of this pull request.

@nicolas2bert
nicolas2bert force-pushed the bugfix/CLDSRV-961/lc-phd branch from 42ec3b9 to ca7fa09 Compare August 13, 2026 10:18
@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
9714 2 9712 0
View the full list of 3 ❄️ flaky test(s)
"after each" hook for "should batch delete 1000 objects quietly"::Multi-Object Delete Success "after each" hook for "should batch delete 1000 objects quietly"

Flake rate in main: 100.00% (Passed 0 times, Failed 2 times)

Stack Traces | 6.96s run time
The bucket you tried to delete is not empty.
"before each" hook for "should batch delete 1000 objects quietly"::Multi-Object Delete Success "before each" hook for "should batch delete 1000 objects quietly"

Flake rate in main: 100.00% (Passed 0 times, Failed 2 times)

Stack Traces | 0.423s run time
socket hang up
should create a bunch of objects and their versions::put and head object with versioning With v4 signature on versioning suspended then enabled bucket w/ null version should create a bunch of objects and their versions

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 600s run time
Timeout of 600000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (.../test/versioning/objectHead.js)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

bucketd runs the lifecycle listing, not cloudserver. The s3c legs therefore
need the fix in the MetaData image, not only in the pinned arsenal.

This points at a build of MetaData bugfix/MD-1359/lc-phd. Repin on a
released tag when MD-1359 ships.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants