Skip to content

replicaCount above 1 runs several un-elected copies of the operators #236

Description

@erkiaas

replicaCount is a plain chart value, but the operators do no leader election —
so raising it above 1 runs several controllers reconciling the same resources,
and there is nothing in the chart's RBAC to coordinate with even if one wanted
to.

Where

src/app.js:50-63 starts every operator unconditionally on boot:

export async function startOperators(provider) {
    const kubeClientOperator = new KubeOIDCClientOperator(provider)
    await kubeClientOperator.watchClients()
    const kubeMiddlewareClientOperator = new KubeOIDCMiddlewareClientOperator(provider)
    await kubeMiddlewareClientOperator.watchClients()
    const kubeUserOperator = new KubeOidcUserOperator(provider)
    await kubeUserOperator.watchUsers()
    const kubeUserEventHookOperator = new KubeOidcUserEventHookOperator()
    await kubeUserEventHookOperator.watchUsers()
    const scimConnectionOperator = new KubeScimConnectionOperator()
    await scimConnectionOperator.watchConnections()
    activityTracker.start()
}

These are writers, not just watchers — they patch OIDCClient / OIDCUser
status, create and update the generated client Secrets, and (via
secretRefreshJobSpec) create Jobs. charts/passmower/templates/deployment.yaml:14
passes replicas: {{ .Values.replicaCount }} straight through, and
charts/passmower/templates/serviceaccount.yaml grants no
coordination.k8s.io leases, so there is no lock available to opt into
either.

Impact

Session state is in Redis and signing keys are in a Secret, so the HTTP side of
passmower looks horizontally scalable and replicaCount: 3 installs happily.
What actually happens is N controllers racing on the same CRs: duplicated status
patches, duplicated secret-refresh Jobs, and activityTracker flushes from
every replica.

The practical consequence is that a single-replica ceiling is a real constraint
rather than a default, and it is not discoverable from the values file. An
authorization server is exactly the component where an operator would want two
replicas — a rolling restart or a lost node currently means nobody can start a
new sign-in until the pod is back.

Suggestions

Either would help; the first is much cheaper:

  1. Document it. A comment on replicaCount in values.yaml saying the
    operators are not leader-elected and the supported value is 1. Cheap, and it
    stops people finding out the interesting way.
  2. Leader-elect the operator loop. A coordination.k8s.io Lease held by one
    replica, with the others serving HTTP only. That separates the two concerns —
    the stateless authorization endpoints could then scale while reconciliation
    stays single-writer.

Happy to be told option 2 is out of scope; the gap between what replicaCount
implies and what is safe seems worth closing one way or the other.

Version

2.2.0 / master.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions