replicaCount is a plain chart value, but the operators do no leader election —
so raising it above 1 runs several controllers reconciling the same resources,
and there is nothing in the chart's RBAC to coordinate with even if one wanted
to.
Where
src/app.js:50-63 starts every operator unconditionally on boot:
export async function startOperators(provider) {
const kubeClientOperator = new KubeOIDCClientOperator(provider)
await kubeClientOperator.watchClients()
const kubeMiddlewareClientOperator = new KubeOIDCMiddlewareClientOperator(provider)
await kubeMiddlewareClientOperator.watchClients()
const kubeUserOperator = new KubeOidcUserOperator(provider)
await kubeUserOperator.watchUsers()
const kubeUserEventHookOperator = new KubeOidcUserEventHookOperator()
await kubeUserEventHookOperator.watchUsers()
const scimConnectionOperator = new KubeScimConnectionOperator()
await scimConnectionOperator.watchConnections()
activityTracker.start()
}
These are writers, not just watchers — they patch OIDCClient / OIDCUser
status, create and update the generated client Secrets, and (via
secretRefreshJobSpec) create Jobs. charts/passmower/templates/deployment.yaml:14
passes replicas: {{ .Values.replicaCount }} straight through, and
charts/passmower/templates/serviceaccount.yaml grants no
coordination.k8s.io leases, so there is no lock available to opt into
either.
Impact
Session state is in Redis and signing keys are in a Secret, so the HTTP side of
passmower looks horizontally scalable and replicaCount: 3 installs happily.
What actually happens is N controllers racing on the same CRs: duplicated status
patches, duplicated secret-refresh Jobs, and activityTracker flushes from
every replica.
The practical consequence is that a single-replica ceiling is a real constraint
rather than a default, and it is not discoverable from the values file. An
authorization server is exactly the component where an operator would want two
replicas — a rolling restart or a lost node currently means nobody can start a
new sign-in until the pod is back.
Suggestions
Either would help; the first is much cheaper:
- Document it. A comment on
replicaCount in values.yaml saying the
operators are not leader-elected and the supported value is 1. Cheap, and it
stops people finding out the interesting way.
- Leader-elect the operator loop. A
coordination.k8s.io Lease held by one
replica, with the others serving HTTP only. That separates the two concerns —
the stateless authorization endpoints could then scale while reconciliation
stays single-writer.
Happy to be told option 2 is out of scope; the gap between what replicaCount
implies and what is safe seems worth closing one way or the other.
Version
2.2.0 / master.
replicaCountis a plain chart value, but the operators do no leader election —so raising it above 1 runs several controllers reconciling the same resources,
and there is nothing in the chart's RBAC to coordinate with even if one wanted
to.
Where
src/app.js:50-63starts every operator unconditionally on boot:These are writers, not just watchers — they patch
OIDCClient/OIDCUserstatus, create and update the generated client Secrets, and (via
secretRefreshJobSpec) create Jobs.charts/passmower/templates/deployment.yaml:14passes
replicas: {{ .Values.replicaCount }}straight through, andcharts/passmower/templates/serviceaccount.yamlgrants nocoordination.k8s.ioleases, so there is no lock available to opt intoeither.
Impact
Session state is in Redis and signing keys are in a Secret, so the HTTP side of
passmower looks horizontally scalable and
replicaCount: 3installs happily.What actually happens is N controllers racing on the same CRs: duplicated status
patches, duplicated secret-refresh Jobs, and
activityTrackerflushes fromevery replica.
The practical consequence is that a single-replica ceiling is a real constraint
rather than a default, and it is not discoverable from the values file. An
authorization server is exactly the component where an operator would want two
replicas — a rolling restart or a lost node currently means nobody can start a
new sign-in until the pod is back.
Suggestions
Either would help; the first is much cheaper:
replicaCountinvalues.yamlsaying theoperators are not leader-elected and the supported value is 1. Cheap, and it
stops people finding out the interesting way.
coordination.k8s.ioLease held by onereplica, with the others serving HTTP only. That separates the two concerns —
the stateless authorization endpoints could then scale while reconciliation
stays single-writer.
Happy to be told option 2 is out of scope; the gap between what
replicaCountimplies and what is safe seems worth closing one way or the other.
Version
2.2.0 /
master.