I see errors like:

2025/06/16 08:43:19 Received error: invalid key: API key does not exist
2025/06/16 08:43:19 control: doLogin(regen=false, hasUrl=false)
2025/06/16 08:43:19 control: RegisterReq: onode= node=[jXY1G] fup=false nks=false
boot: 2025/06/16 08:43:44 failed to read from tailscaled: context deadline exceeded

in production and staging. There was some claim on friday that maybe this was due to @Ollie.Shotton_WMDE running make apply while missing some tailscale credentials locally but as I understand it this shouldn't matter and the key shouldn't end up updated.

on staging I ran:

$ k get secret -n tailscale operator-oauth --show-managed-fields -o jsonpath='{range .metadata.managedFields[*]}{.manager}{" did "}{.operation}{" at "}{.time}{"\n"}{end}'
HashiCorp did Update at 2024-04-18T08:46:35Z

which to me suggests that the secret hasn't been altered since creation last year.

My gut feeling is that this is due to an old tailscale operator that now has some expired key somewhere. I'll trying "just" bumping to the latest version of the tailscale operator on staging.

Confirmed fixed after merging: https://github.com/wmde/wbaas-deploy/pull/2165 and https://github.com/wmde/wbaas-deploy/pull/2166. Clearly a sign we need to be more diligent in keeping the operator up to date.