Add admission webhook TLS mismatch problem - #777

Adds a new SREGym problem simulating an admission-webhook TLS trust mismatch in the Hotel Reservation app. The failure is a production-style Kubernetes admission-control issue: the webhook backend is reachable, but the ValidatingWebhookConfiguration contains a stale/wrong caBundle, so the kube-apiserver rejects pod CREATE requests with an x509 verification error.

The PR is limited to the new problem plus registry wiring.

1. The real-world failure story

Admission webhooks sit directly in the Kubernetes API request path. If a ValidatingWebhookConfiguration has failurePolicy: Fail, then matching requests are rejected whenever the webhook call fails. This is risky because the affected application can be healthy, but Kubernetes may still be unable to create replacement pods.

This problem models a certificate-rotation / CA-drift failure: the webhook backend is reachable and serving HTTPS, but the caBundle stored in the ValidatingWebhookConfiguration is stale or wrong. The kube-apiserver therefore cannot verify the webhook server certificate and rejects pod CREATE admission requests with an x509 error.

Several real-world reports anchor this simulation:

• OPA cert-controller, October 2020 — “CA and Server certificate potentially get updated before ValidatingWebhookConfiguration”. The issue describes the risk this problem models: the CA is renewed, but the ValidatingWebhookConfiguration may still contain the old CA, so webhook calls can fail.

• ingress-nginx, August 2020 — “apply ingress rule error after install ingress-nginx: x509 certificate is not valid”. A validating admission webhook rejects resource creation because the serving certificate does not validate for the webhook service DNS name.

• ingress-nginx, January 2022 — “Kubernetes: ingress-nginx-controller-admission error, x509 certificate signed by unknown authority”. The writeup shows an admission-controller x509 failure after reinstalling ingress-nginx, with a workaround that patches the validating webhook caBundle.

• cert-manager, September 2023 — “Webhook inject-ca-from annotation causes downtime”. The report describes a rotation race where the API server accepts one certificate chain while the webhook server still serves another, causing webhook downtime.

The SREGym scenario keeps the same failure class but makes it reproducible: the backend exists and is reachable, while the trust relationship between the kube-apiserver and webhook is intentionally broken.

2. How the failure is simulated in SREGym

The new problem class lives at:

sregym/conductor/problems/admission_webhook_tls_mismatch.py

and is registered as:

admission_webhook_tls_mismatch_hotel_reservation

The fault injection does the following:

1. Creates a temporary policy-system namespace.
2. Generates TLS material:
   • one CA that signs the webhook server certificate,
   • a separate wrong CA used as the stale caBundle.
3. Deploys a small HTTPS webhook backend as policy-system/pod-policy-webhook.
4. Installs a ValidatingWebhookConfiguration named pod-policy.validation.k8s.io.
5. Scopes the webhook to the target app namespace using namespaceSelector.
6. Configures the webhook with:
   • failurePolicy: Fail
   • pod CREATE admission rules
   • the reachable webhook service
   • the intentionally wrong caBundle
7. Deletes one pod from the recommendation deployment so the ReplicaSet has to recreate it.

The recreate attempt then hits the webhook and fails during TLS verification.

3. Problem runtime behaviour

Verified locally through the SREGym CLI and Kubernetes events.

The key event observed locally:

failed calling webhook "pod-policy.validation.k8s.io":
tls: failed to verify certificate:
x509: certificate signed by unknown authority

This is the main diagnostic signal an agent should follow. The pod is not failing because of its own image, command, memory, or service config; pod creation is being rejected by admission control.

4. Mitigation and validation

Valid mitigations:

• delete the broken ValidatingWebhookConfiguration,
• patch failurePolicy from Fail to Ignore,
• or repair the webhook trust chain by restoring the correct caBundle.

Local mitigation test used the first path:

kubectl delete validatingwebhookconfiguration pod-policy.validation.k8s.io

After that, the ReplicaSet recreated the missing recommendation pod and the SREGym mitigation oracle passed.

Local final result:

5. Expected agent behaviour

I have not run Stratus end-to-end on this problem yet.

Expected successful agent path:

1. Notice recommendation is under-replicated.
2. Inspect the ReplicaSet / events instead of only checking the application pod logs.
3. Find that pod creation is failing at admission time.
4. Identify the cluster-scoped ValidatingWebhookConfiguration.
5. Distinguish a TLS trust mismatch from a missing webhook backend.
6. Mitigate by deleting the webhook, changing failurePolicy, or repairing caBundle.

The intended challenge is that the visible symptom is application-level availability loss, but the root cause is a cluster-scoped admission-control dependency.

Notes for reviewers

• No new Python package dependency is added.
• The webhook backend is created dynamically as a small Python HTTPS server.
• The failure is scoped to the target app namespace using namespaceSelector.
• Recovery deletes both the webhook configuration and the temporary policy-system namespace.
• Currently only the Hotel Reservation variant is registered; sibling variants for social_network or astronomy_shop can be added if useful.