Reconcile the Scaler and propagate replicas to the StatefulSet #873

Part of #872

Problem

We added the Scaler CRD to operator-rs in 0.110.0.
Now we need to make it useful :)

Scope

This issue is not about making autoscaling fully work or even "safe".
That comes later on top of this.

The scope of this is purely:

1. Ship the CRD: commons-operator installs Scaler the same way it already installs AuthenticationClass et. al. including the conversion webhook stuff

2. Extend replicas in the product CRDs so a role group can declare that something else owns its replica count.

"Pseudocode":

roleGroups:
  default:
    replicas: 3              # unchanged

roleGroups:
  default:
    replicas:
      externallyScaled: {}   # operator creates a Scaler, something else sets the number (that bit is out of scope)

roleGroups:
  default:
    replicas:
      externallyScaled:
        min: 2
        max: 10

All of this is up-to-decision. I never know what we can do with complex enums and so on so I'll leave the CRD decision to the implementer.
This or something like it was proposed by @soenkeliebau in stackabletech/operator-rs#1181.
I'm not sure how much of that design was thought through vs. AI generated.

Further tickets will require additions here for "real" autoscaling. So we need to consider a "auto" variant.
And the normal "fixed" replica count obviously also needs to continue working.

I'm pretty sure we want to support bounds (min/max) somewhere. I'm just not sure if they belong on the Scaler or in the product CRDs. Another open question.

3. A reconcile function in operator-rs that a product operator calls. Reads spec.replicas from the Scaler, writes it to the StatefulSet, writes status.replicas and status.selector back. Here I'm super unsure how it fits with the smooth-operator team. So check with them if this makes sense :)

One thing Claude flagged during the review of Scaler and I don't know if there is a good reason for this:
status.selector is an Option<String> and an absent selector does not fail.
The HPA controller counts ready pods matching that selector and multiplies that number by its computed ratio so a selector matching nothing produces a desired count of zero.
If that is correct then I agree with Claude that this should be required rather than optional.
But again maybe @soenkeliebau and @Techassi had a good reason.

4. Wire up one product so this is testable end to end.

5. When that has been proven and all decisions have been finalised: Roll this out everywhere

Out of scope

• Nothing with any state machine, no guard, hook (see Epic: Safe and automatic scaling #872 ). This ticket only ever leaves the Scaler in Idle.
• We don't build or integrate anything that actually writes scaler.spec.replicas -> That is for later. For testing a HPA can be used.
• Any per-product drain work.
• Creating HPAs

Open questions

• Who reconciles the Scaler? Sönke's experimental PR put it in the nifi-operator. I think that makes sense. Right now there are no product-specific things we do but later we might wan tto.
• Which product first? Hive could be an easy one: Stateless and single-role so we don't have to do anything with draining. OTOH Trino or NiFi would be more useful in the end but also more work. Hence my preference would be for Hive. Let me know if you want to deviate from that.
• CRD shape
• selector -> Optional or not?
• What replica count does the StatefulSet start at if someone sets externallyScaled and nothing ever writes to the Scaler? Does the operator seed it, and with what?
• Where do bounds go?

Expected effort

8 days, if it takes longer, stop and tell me.
My reasoning: I hope that most of the time would be spent on decisions and design and later testing and that it's not actually a lot of code and the code that is needed can be mostly AI generated.
The rollout follows existing patterns.

Notes

operator-rs#1181 may be wroth a look. It contains parts of this issue (outdated etc. but the ideas are partially there).
It uses StackableScaler and generates HPAs which I explicitly do NOT want us to do.