squid: mgr, osd, mon: remove pg_upmap_primary mappings for PG merge targets - #71051
Open
ljflores wants to merge 5 commits into
Open
squid: mgr, osd, mon: remove pg_upmap_primary mappings for PG merge targets#71051ljflores wants to merge 5 commits into
ljflores wants to merge 5 commits into
Conversation
Now, the command `ceph osd rm-pg-upmap-primary-all` can optionally be run for a specific pool. The syntax is: `ceph osd rm-pg-upmap-primary-all <pool name>` This command can be used in a developer or advanced user setting, but it is also helpful in the DaemonServer code when we remove mappings on pools where pg_num was adjusted. I also clarified the descriptions in the pg-upmap-primary CLI commands. Fixes: https://tracker.ceph.com/issues/76731 Signed-off-by: Laura Flores <lflores@ibm.com> (cherry picked from commit f4beec9)
`OSDMap::check_pg_upmaps` already takes care of removing pg_upmap_primary mappings for: 1. PGs that were “merge sources” (PGs that merged into another PG and thus no longer exist in the pool) 2. PGs that are “pending merge” (these PGs will end up becoming merge sources) But there is a third scenario that was missing for pg_upmap_primary: “merge targets”. The DaemonServer identifies PGs as “merge targets”, or the PGs that a “merge source” will merge into. If there is a pg_upmap, pg_upmap_items, or pg_upmap_primary mapping associated with a “merge target”, the DaemonServer issues this message: ``` 2026-05-14T21:16:56.700+0000 7f745e33d640 10 mgr.server operator() pool 5 pg_num_target 32 pg_num 33 - merge target 5.0 acting does not match (source [3,2,1] != target [3,1,2]) ``` Until the acting set of the “merge target” matches the “merge source”, the merging process will stall. One way the PG merging process is triggered is by removing the “bulk” flag from a pool. Removing upmap mappings for “merge target” PGs is not handled in the OSDMap code, but rather, the DaemonServer code. This patch adds in that missing logic. Fixes: https://tracker.ceph.com/issues/76731 Signed-off-by: Laura Flores <lflores@ibm.com> (cherry picked from commit 460b2c8)
Add test_read_balancer.sh to the rados teuthology workunits. The script is used to test any kind of balancer operations, whether it be via the mgr module or the CLI upmap commands. TEST_read_balancer_bulk_flag tests the balancer's interaction with the autoscaler --bulk flag and ensures that pg_upmap_primary mappings are properly cleared when a pool grows and reduces its pg_num. TEST_read_balancer_cli verifies the new CLI option to remove pg_upmap_primary mappings for a single pool: `ceph osd rm-pg-upmpa-primary-all <pool>`. Also, add a teuthology suite yaml to singleton-nomsgr. The `test_read_balancer.sh` script does not tolerate msgr error injections, so this is the best location for it as opposed to the singleton tests. Fixes: https://tracker.ceph.com/issues/76731 Signed-off-by: Laura Flores <lflores@ibm.com> (cherry picked from commit 09eb9bd)
There is now the option to remove all pg_upmap_primary mappings on a single pool. Fixes: https://tracker.ceph.com/issues/76731 Signed-off-by: Laura Flores <lflores@ibm.com> (cherry picked from commit 6f99c8d)
…rations This test intentially made a test pool scale up and down (via the bulk flag) so we could verify that the read balancer's `pg_upmap_primary` entires get properly removed as PGs merge. Unintentionally, the test caused slow heartbeats because of a large amount of PG splitting that was happening as the pool scaled up. A large amount of PG splitting was causing the op threadpool to be consumed for up to 7 minutes, which exceeded the default osd_op_thread_timeout of 15 seconds and led to slow heartbeats. Adjusting `mgr_max_pg_num_change` from 128 to 32 means that there will be less PG splitting per osdmap epoch, and therefore less consumption on the op threadpool. Now, it takes only ~7 seconds for PG splitting per epoch. I also adjusted the `osd_op_thread_timeout` to 60s since, although much improved, the latest results still showed 2/30 tests failing from slow heartbeats. See full analysis of this issue here: https://gist.github.com/ljflores/72e66c405aaf2d3e004eddcbc77545bf Fixes: https://tracker.ceph.com/issues/76731 Signed-off-by: Laura Flores <lflores@ibm.com> (cherry picked from commit 543e9e6)
Member
Author
|
jenkins test make check |
rzarzynski
approved these changes
Aug 14, 2026
rzarzynski
left a comment
Contributor
There was a problem hiding this comment.
LGTM – straightforward cherry-picks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backport tracker: https://tracker.ceph.com/issues/79487
backport of #69272
parent tracker: https://tracker.ceph.com/issues/76731
this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh