Skip to content

squid: mgr, osd, mon: remove pg_upmap_primary mappings for PG merge targets - #71051

Open
ljflores wants to merge 5 commits into
ceph:squidfrom
ljflores:wip-79487-squid
Open

squid: mgr, osd, mon: remove pg_upmap_primary mappings for PG merge targets#71051
ljflores wants to merge 5 commits into
ceph:squidfrom
ljflores:wip-79487-squid

Conversation

@ljflores

Copy link
Copy Markdown
Member

backport tracker: https://tracker.ceph.com/issues/79487


backport of #69272
parent tracker: https://tracker.ceph.com/issues/76731

this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh

Now, the command `ceph osd rm-pg-upmap-primary-all` can optionally be
run for a specific pool. The syntax is:

`ceph osd rm-pg-upmap-primary-all <pool name>`

This command can be used in a developer or advanced user setting,
but it is also helpful in the DaemonServer code when we remove
mappings on pools where pg_num was adjusted.

I also clarified the descriptions in the pg-upmap-primary CLI
commands.

Fixes: https://tracker.ceph.com/issues/76731
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit f4beec9)
`OSDMap::check_pg_upmaps` already takes care of removing pg_upmap_primary
mappings for:

1. PGs that were “merge sources” (PGs that merged into another PG and thus
   no longer exist in the pool)
2. PGs that are “pending merge” (these PGs will end up becoming merge sources)

But there is a third scenario that was missing for pg_upmap_primary: “merge targets”.

The DaemonServer identifies PGs as “merge targets”, or the PGs that a “merge source”
will merge into. If there is a pg_upmap, pg_upmap_items, or pg_upmap_primary mapping
associated with a “merge target”, the DaemonServer issues this message:

```
2026-05-14T21:16:56.700+0000 7f745e33d640 10 mgr.server operator() pool 5 pg_num_target
32 pg_num 33 - merge target 5.0 acting does not match (source [3,2,1] != target [3,1,2])
```

Until the acting set of the “merge target” matches the “merge source”, the merging process
will stall. One way the PG merging process is triggered is by removing the “bulk” flag from a pool.

Removing upmap mappings for “merge target” PGs is not handled in the OSDMap code, but rather, the
DaemonServer code. This patch adds in that missing logic.

Fixes: https://tracker.ceph.com/issues/76731
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 460b2c8)
Add test_read_balancer.sh to the rados teuthology workunits.
The script is used to test any kind of balancer operations, whether it
be via the mgr module or the CLI upmap commands.

TEST_read_balancer_bulk_flag tests the balancer's interaction with the
autoscaler --bulk flag and ensures that pg_upmap_primary mappings are
properly cleared when a pool grows and reduces its pg_num.

TEST_read_balancer_cli verifies the new CLI option to remove pg_upmap_primary
mappings for a single pool: `ceph osd rm-pg-upmpa-primary-all <pool>`.

Also, add a teuthology suite yaml to singleton-nomsgr. The `test_read_balancer.sh`
script does not tolerate msgr error injections, so this is the best
location for it as opposed to the singleton tests.

Fixes: https://tracker.ceph.com/issues/76731
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 09eb9bd)
There is now the option to remove all pg_upmap_primary mappings
on a single pool.

Fixes: https://tracker.ceph.com/issues/76731
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 6f99c8d)
…rations

This test intentially made a test pool scale up and down (via the bulk flag)
so we could verify that the read balancer's `pg_upmap_primary` entires get
properly removed as PGs merge.

Unintentionally, the test caused slow heartbeats because of a large amount of
PG splitting that was happening as the pool scaled up. A large amount of PG
splitting was causing the op threadpool to be consumed for up to 7 minutes,
which exceeded the default osd_op_thread_timeout of 15 seconds and led to
slow heartbeats.

Adjusting `mgr_max_pg_num_change` from 128 to 32 means that there will be less PG
splitting per osdmap epoch, and therefore less consumption on the op threadpool.
Now, it takes only ~7 seconds for PG splitting per epoch.

I also adjusted the `osd_op_thread_timeout` to 60s since, although much
improved, the latest results still showed 2/30 tests failing from slow heartbeats.

See full analysis of this issue here: https://gist.github.com/ljflores/72e66c405aaf2d3e004eddcbc77545bf

Fixes: https://tracker.ceph.com/issues/76731
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 543e9e6)
@ljflores
ljflores requested review from a team as code owners August 13, 2026 20:51
@ljflores ljflores added this to the squid milestone Aug 13, 2026
@ljflores ljflores added the core label Aug 13, 2026
@github-actions github-actions Bot added the releng-audit-pass Release engineering: passed backport verification audit. label Aug 13, 2026
@ljflores

Copy link
Copy Markdown
Member Author

jenkins test make check

@rzarzynski rzarzynski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM – straightforward cherry-picks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core needs-qa releng-audit-pass Release engineering: passed backport verification audit. wip-lflores-testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants