Repository navigation
tentacle: mds: reset heartbeat in cache-wide walks during resolve and rejoin - #73074
Open
anthonyeleven wants to merge 2 commits into
Open
anthonyeleven wants to merge 2 commits into
anthonyeleven wants to merge 2 commits into
Conversation
recalc_auth_bits() (end of up:resolve) and rejoin_scour_survivor_replicas() (handling a weak rejoin, on both a rejoining rank and a survivor) walk the entire cache under mds_lock without resetting the MDS internal heartbeat. With a large cache (e.g. after replaying a very long journal) these walks take long enough for the heartbeat to time out: heartbeat_map is_healthy 'MDSRank' had timed out after 15.000000954s mds.beacon.X Skipping beacon heartbeat to monitors (last acked 4s ago); MDS internal heartbeat is not healthy! The mons then mark the MDS laggy and, with a standby available, fail the rank -- which restarts recovery and hits the same walk again. Reset the heartbeat every heartbeat_reset_grace() items, as is done elsewhere on the recovery path (e.g. trim_unlinked_inodes()). Fixes: https://tracker.ceph.com/issues/81449 Related-to: https://tracker.ceph.com/issues/64717 Reported-by: Dan van der Ster <dan.vanderster@clyso.com> Reported-by: Abhishek Lekshmanan <abhishek.lekshmanan@cern.ch> Signed-off-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit 2be25a8)
Besides recalc_auth_bits() and rejoin_scour_survivor_replicas(), several other recovery paths walk an unbounded part of the cache, or process a message built by such a walk, under mds_lock without resetting the MDS internal heartbeat: - rejoin_walk(): on a survivor, walks every replica it holds of a rejoining rank's subtrees to build the strong rejoin; on a rejoining rank, builds the weak rejoin. - rejoin_send_acks(): on a rejoining rank, walks every auth item in the cache. - handle_mds_recovery(): on a survivor, walks every replica it holds of the recovered rank's subtrees to collect waiters. - handle_cache_rejoin_strong() and handle_cache_rejoin_ack(): process the messages built by rejoin_walk() and rejoin_send_acks(), so they are as large as those walks. rejoin_walk() and handle_mds_recovery() run on active ranks, so a heartbeat timeout there can get a healthy survivor marked laggy and failed as well. Reset the heartbeat every heartbeat_reset_grace() items, as done for the other recovery walks. rejoin_walk() is recursive, so the counter is passed down from rejoin_send_rejoins(). Fixes: https://tracker.ceph.com/issues/81449 Suggested-by: Dan van der Ster <dan.vanderster@clyso.com> Signed-off-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit e9fdf5c)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
github-actions
Bot
added
the
releng-audit-fail
anthonyeleven
marked this pull request as ready for review
October 25, 2026 14:27
Contributor
Author
|
/audit retest |
|
🔄 Audit Retest Triggered by @anthonyeleven. |
github-actions
Bot
added
releng-audit-pass
Contributor
Author
|
jenkins test docs |
github-actions
Bot
added
the
releng-audit-pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backport tracker: https://tracker.ceph.com/issues/82070
backport of #72414
parent tracker: https://tracker.ceph.com/issues/81449
@vshankar this one affects our production clusters. A rank in resolve or rejoin with a large cache misses its heartbeat during these walks, gets marked laggy and failed, and each replacement repeats the same walk. We are moving our fleet to tentacle, so I would like to see this in the earliest tentacle and umbrella releases possible. Clean cherry-pick of both commits from #72414, no follow-up fixes on main.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an
xbetween the brackets:[x]. Spaces and capitalization matter when checking off items this way.Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.
Assisted-by: Claude Opus-5.5