I have an idea.
We can mark all classes which should not used by users as @internal (https://docs.phpdoc.org/2.9/references/phpdoc/tags/internal.html) and make a new patch release.

After that, we will be able to change them as we want without fear of violating B/C.

What do you think?

But I'll need you help here as I don't know which classes must remain "public".