Add admission_webhook_outage_hotel_reservation problem

Adds a new problem simulating an admission-webhook outage — a well-attested production failure class not currently covered by the SREGym catalogue. Includes a minimal mcp-server RBAC expansion (9 lines of YAML) needed for agents to mitigate this class of failure; rationale below. The PR can be landed as one piece or split (the problem alone is still a legitimate benchmark result under the default RBAC; see Stratus Run 1).

1. The real-world failure story

When a ValidatingWebhookConfiguration with failurePolicy: Fail points at a backend whose endpoints are missing or unreachable, every admission request the webhook intercepts is rejected by the kube-apiserver. Because admission control sits in the request path for every matching CREATE, the cluster (or matching namespaceSelector scope) loses the ability to create pods even though deployments, images, and nodes are all healthy. Recovery requires an operator with cluster-scoped permissions on admissionregistration.k8s.io.

Three public postmortems anchor this simulation, all listed on k8s.af (the failure-story aggregator referenced in the SREGym evaluation task spec):

• Jetstack, 2019 — "How a simple admission webhook lead to a cluster outage." A ValidatingWebhookConfiguration on a GKE cluster interacted badly with GKE node auto-repair, causing prolonged downtime and a failed master upgrade. (Original blog redirects through Venafi → CyberArk post-acquisition; k8s.af is the canonical pointer.)
• Wyssmann Engineering, December 2022 — "OPA Gatekeeper and issue while doing a cluster restore". Both Gatekeeper webhook configurations blocked pod lifecycle operations across CoreDNS, Canal, metrics-server, and ingress. Resolution required literally kubectl delete ValidatingWebhookConfiguration plus the mutating counterpart — the same mitigation surface our problem expects.
• Airbnb, 2020 KubeCon — "10 More Weird Ways to Blow Up Your Kubernetes". Covers a MutatingAdmissionWebhook failure mode among other things.

Recurring upstream issues confirm the class is ongoing: kubernetes/kubernetes#80313 (dead-TCP-affected webhooks) and cert-manager/cert-manager#5870 (cert-manager webhook gateway timeouts).

2. How the failure is simulated on SREGym

New problem class at sregym/conductor/problems/admission_webhook_outage.py. inject_fault installs a cluster-scoped ValidatingWebhookConfiguration with failurePolicy: Fail and a namespaceSelector scoped to hotel-reservation, whose clientConfig.service points at a service in default/ that does not exist. It then deletes one pod from the recommendation deployment so the ReplicaSet's recreate attempt hits the broken webhook and is rejected. caBundle is intentionally omitted so the apiserver uses its system trust roots and the failure surfaces at the backend-lookup step (service not found), matching the documented narrative rather than a cert-parsing error.

The problem is registered as "admission_webhook_outage_hotel_reservation" in the DIRECT K8S API section of registry.py. The class is parameterised over app_name (hotel_reservation / social_network / astronomy_shop) so sibling variants are a one-line follow-up.

recover_fault is idempotent: if the agent already removed the webhook during mitigation, recovery logs already absent and exits cleanly. The conductor's existing baseline reconcile (validating_webhook_configs_deleted) verifies no leftovers either way. The generic MitigationOracle accepts all three valid mitigation paths — delete the webhook, patch failurePolicy to Ignore, or restore the backend service — since each results in pod health.

3. Problem runtime behaviour

Verified across 5 end-to-end runs on a 4-node ARM kind cluster (1 control-plane + 3 workers, 16 GiB Docker).

A clean human-driven run (me, via the CLI) scored 100/100 on diagnosis (D1=D2=D3=1.0, all 9 judge checklist questions Yes), mitigation passed, TTL=181.7 s, TTM=228.9 s. Back-to-back start cycles work without manual intervention; the baseline reconcile handles even mid-mitigation abnormal exits.

4. Agent behaviour (Stratus, GPT-5)

Ran Stratus twice — both runs are informative.

The D2 nit in both runs is the judge preferring "present with no endpoints" over "not found" — both are technically true of this fault.

Run 1 is itself a research finding. It demonstrates that admission-webhook outages of this form cannot be self-mitigated by an agent restricted to namespace-scoped workload RBAC, mirroring the real-world property that on-call engineers typically need a break-glass admin kubeconfig to recover from these incidents.

Run 2 is a separate finding worth flagging to the Stratus team: when ClusterStateOracle cannot reach 127.0.0.1:16443 after a successful submit, the retry/rollback heuristic produces an outcome strictly worse than accepting the submission — it actively tries to undo correct mitigations. In our case the RBAC scope happened to prevent the rollback from succeeding, which was fortunate.

RBAC change rationale

mcp_server/k8s/clusterrole.yaml adds one rule:

- apiGroups: ["admissionregistration.k8s.io"]
  resources:
    - validatingwebhookconfigurations
    - mutatingwebhookconfigurations
  verbs: ["get", "list", "watch", "delete", "patch", "update"]

get/list/watch lets the agent discover that a webhook is the root cause; delete/patch/update lets it execute any of the three valid mitigations. create is deliberately excluded — the framework controls webhook installation via inject_fault, and an agent re-creating a faulty webhook to "undo" a correct mitigation is exactly the pathological behaviour Run 2 illustrated. This mirrors the least-privilege scope an on-call SRE would expect on a temporary elevated kubeconfig.

If the maintainers prefer to land the RBAC change separately, the problem file can ship as-is — Run 1's TIMEOUT is itself a legitimate benchmark floor.

Notes for reviewers

• No new Python dependencies; uses the already-imported kubernetes client.
• Follows the NetworkPolicyBlock style (dict-shaped resource bodies) to avoid version-skew on typed-class names across kubernetes-client releases.
• Sibling variants for social_network / astronomy_shop are a one-line follow-up; happy to add in a separate PR if useful.
• Open to feedback on whether the dual-Stratus-run detail belongs here or in a separate research artifact.