What problem does this address?

At the moment, `remove_theme_support( 'core-block-patterns' ); is an all or nothing with allowing pattern directory patterns to be viewed in the inserter.
Being able to still use the theme.json registration of pattern via slugs from the pattern directory, allows a curated list of patterns and more granular control.

What is your proposed solution?

The theme developer would set in functions.php remove_theme_support( 'core-block-patterns' );
and set in theme.json the list of pattern allowed: "patterns": [ "short-text-surrounded-by-round-images", "partner-logos" ]

There could be a check that if there is a theme.json pattern setting, it would allow them to be visible in the Inserter.
Instead an "Disallow all" it should be an explicit allow.

With the growing number of pattern in the directory, a curation would help with the exposure of patterns in the editor. The more patterns there are the more theme developer would want to restrict the visiblity. This feature would give them the means instead of just blocking all patterns from the directory, select the few that should work well.