Add mutating_webhook_resource_limits problem - #847
Conversation
|
@keremakcali Can you please link some real-world incident stories that inspired this fault? Not generic troubleshooting guides. Preferably post-mortems. Without an incident to ground it in, it's harder to judge the relevance of a fault. Although I do love that the diagnosis for this fault is not quite straight forward! |
|
Oh, I'm so sorry. I pasted the wrong reference link. I unfortunately don't have an official post-mortem because I had a hard time finding one that wasn't already implemented or mentioned in posted RFCs, and that I could understand well enough to believe I could implement something reliably (or i am bad at googling). So I'm using this GitHub issue as the basis of this implementation instead. Also, I just updated the PR text to better reflect how i decided to go through with this implementation. I hope it's alright @Saadmrp1038 . |
|
@keremakcali That's alright. I will need to review to see how the agent performs on this fault over multiple runs. It seems good at first glance. Maybe we can tweak some stuff to make it harder. |
@Saadmrp1038 Sounds great! Thanks a lot for the reply. |
Removed incorrect information about mitigation options.
There was a problem hiding this comment.
@keremakcali Thanks for the effort! The implementation looks good. I tested it end-to-end and everything works correctly.
One thing is as it is now the problem is too easy. There are a couple of tweaks that can be done to make it harder for the agent to solve:
-
Consider patching the
nginx-thriftdeployment with an explicit memory limit (e.g.256Milimit /128Mirequest) BEFORE activating the webhook, so the deployment spec shows a real value. Right nowresources: {}in the spec immediately signals something injected this field when the pod shows16Mi. The webhook currently only patcheslimits.memory. If you set both requests and limits in the spec, you'll need the webhook to patch both fields (otherwise Kubernetes rejects the pod withrequests > limits). This also feels like a more realistic patch. -
Consider installing 3-4 decoy MutatingWebhookConfigurations (e.g. cert-manager, istio, kyverno names). Although make sure they don't target/affect anything.
Please ping me when you've addressed them.
- Memory limit/request patching before activating the webhook - Decoy pods for increased complexity
|
Hey @Saadmrp1038, I implemented these changes. Tried it out with the agent and it seems like it still can solve it. I tried to address some minor details it had used in the runs as well with not much success. The main thing that makes it easy for the agent seems to be how easy it is to notice a single OOMKilling pod that has a different memory request/limit from its spec. All the other complexities just increase the runtime until it finally finds the correct webhook that mutates it from decoys. I have an idea to make this aspect harder, so I wanted to ask if it makes sense: I was planning to add more pods that get affected by this mutating webhook that injects a very low memory limit. In this case some pods will OOMKill but some will not because they don't require as much memory. I did one such run and the agent did fix the OOMKilling pods but didn't notice the non-OOMKilling pods. Maybe the mitigation oracle should check if it addressed all affected pods. This part is where I am not sure if it makes sense though, because it feels a little bit like an artificially created trap. But at the same time, if those limits persist they might eventually crash due to things like traffic or other stuff that increase their memory usage, which makes them worth checking in the oracle. I would appreciate your feedback on this. |
|
@keremakcali The mitigation is easy that I know. Does it get diagnosis right too? My thought was the agent would see another plausible hypothesis and submit that as root cause without exploring all the options. |
|
@Saadmrp1038 Yeah, actually in diagnosis a lot of the time it frames the nginx-thrift as the faulty component even though it is the victim. I think in one run it thought a decoy MWC was the culprit. Those are the problems that i saw about diagnosis. It almost always figures out the right MWC in the mitigation phase eventually tho. I am not super sure what to get of all these but yeah diagnosis is usually problematic. I was mostly focusing on the mitigation... |
|
@keremakcali Diagnosis failing but mitigation succeeding is also an interesting thing imo. The agent, when diagnosing, initially finds a plausible reason and stops. It doesn't try to make sure there isn't any other possible cause for the issue. But when mitigating, when the mitigation strategy based on the diagnosis fails, the agent then expands its search and finds out that the root cause is somewhere else. That's why previous suggestions were mostly about adding distractors and not about making the mitigation harder. Because I don't think we can make the mitigaiton for this fault that hard anyway. |
|
@Saadmrp1038 Hmm.. That makes a lot of sense. In that case i think i will be pushing the changes and we see what happens. |
- Make decoys realisticly named - Similar minor changes
|
Hi @Saadmrp1038 just pushed the implementation addressing the tweaks. |
|
@keremakcali Looks good to me. Claude Code most of the time fails diagnosis by a few points due to not targeting the webhook configuration itself as the faulty component but rather blaming the backend service (platform-policy-controller). Stratus fails more drastically as it never checks mutating webhooks at all and submits a wrong diagnosis blaming the pod/deployment configuration. Congrats on your first PR merged here 🎉 |
* Add mutating_webhook_resource_limits problem * Add mutating_webhook_resource_limits to Problem List * mutating_webhook_resource_limits problem docstring fix Removed incorrect information about mitigation options. * Additions to mutating_webhook_limits_resource_limits problem - Memory limit/request patching before activating the webhook - Decoy pods for increased complexity * More additions to mutating_webhook problem - Make decoys realisticly named - Similar minor changes --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com>
* Updated default mitigation oracle * Stop timeout for 0 replicas * TBC in CloudLab * Pushing all attempts at the problem * Enable idempotence * Add mutating_webhook_resource_limits problem (#847) * Add mutating_webhook_resource_limits problem * Add mutating_webhook_resource_limits to Problem List * mutating_webhook_resource_limits problem docstring fix Removed incorrect information about mitigation options. * Additions to mutating_webhook_limits_resource_limits problem - Memory limit/request patching before activating the webhook - Decoy pods for increased complexity * More additions to mutating_webhook problem - Make decoys realisticly named - Similar minor changes --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Derive the default problem namespace from the application (#876) * add the the probem redundency * base. * the fixed and dynamic app name are fixed. hte namespace are also fixed. * unessessry formating is reverted. * unessessry formating is reverted. * formating correction. * formating correction. * formating correction. * add fault injection decorator markers to test_problem_bsae.py --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Confluent kafka attempt * Custom image attempt * Threaded producer and added consumer * Syntax error * Added some imports * fix edge cases * run ruff check adn format * New OOM attempt * Force crash on OOM * Add file_descriptor_exhaustion problem (#832) * add file_descriptor_exhaustion problem * Modify fault injection and recovery methods * Fix fault recovery * Change fault injection method Add fault injection by flooding connections * Update fault injection method and add custom oracle --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Finalizer deadlock controller clean (#873) * finalizzer code is reaoslve dadlock benchmark. * yaml file corrected. * reverting hte comments back to their place. * read only rbac * controller log is generalized and avoiding any sregym related name exposing. * applied ruff linter formatter accordingly. * replace metadata naming and includding all the oprations on mitigation side. --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Add: secret_rotation_stale_env_credentials_astronomy_shop problem (#851) * Add secret rotation stale env credentials problem * Simplify secret rotation mitigation oracle checks * Require rotated PostgreSQL in oracle * Fix stale DB credential rotation shortcuts - Remove old password from postgresql-init ConfigMap - Keep PostgreSQL password rotation durable after pod restart - Reject mitigations that revert otelu to the old password - Remove old password leaks from unrelated pods * Fix fault recovery to satisfy the oracle * Fix problem registry import ordering --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Add cumulative_admission_webhook_timeout_hotel_reservation problem (#886) * Add cumulative_admission_webhook_timeout_hotel_reservation problem * Pin webhook backends to worker nodes via nodeAffinity * Make webhook backends realistic admission services Replace the dummy webhook backends with HTTPS admission servers that terminate TLS using Secret-mounted certificates, validate AdmissionReview requests, and return proper admission responses. Certificates are signed by the same CA referenced in each webhook's `caBundle`, with SANs covering the backend Service DNS names. The admission logic enforces a trusted image registry allowlist and a per-container CPU limit. All Hotel Reservation workloads satisfy these policies, so the webhooks remain transparent during normal operation and recovery. Also: * Vary webhook `timeoutSeconds` (12, 8, 11, 9) to make the cumulative timeout less obvious. * Rename the CA CN from `sregym-webhook-ca` to `compliance-webhook-ca`. * Remove a duplicate `rule.from_` check in the oracle. * Fix create-or-replace handling for webhooks, NetworkPolicies, Deployments, and Services so re-injection updates existing resources instead of failing or silently skipping changes. * Model the fault as a missing apiserver NetworkPolicy allow * Model the fault as a missing apiserver allow with a near-miss policy --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Add psa_restricted_blocks_recreation problem (#891) Labelling a namespace pod-security.kubernetes.io/enforce=restricted leaves running pods untouched but makes the apiserver reject recreated pods that run as root / set no seccomp profile, so a deployment silently goes under-replicated. Unlike the existing webhook problems, the policy is enforced by the apiserver via a namespace label, not a webhook config. * Add copilot cli agent (#893) * add copilot cli agent * remove -s flag * Fix MCP retry error handling and CRLF parsing in CI (#896) * fix(mcp): broaden MCP retry exceptions and return error instead of raising * fix(ci): strip CRLF from PR description before parsing problem ID * Fix kind cluster detection (#897) * Fix custom kind cluster detection * Handle failed kind detection output * fix edge cases * run ruff check adn format * Shell state persistence (#903) * Include shell persistence * Include shell persistence * Isolate problem IDs from agent runtime artifacts * update SREGym-applications (18620ed) (#910) * Add Calico route-reflector label drift problem (#828) * Add Calico route-reflector label drift problem * Address Calico route-reflector review comments * Harden Calico cleanup ownership tracking * Fix Codex agent install verification * Tighten Codex platform package detection * Restore kubelet eviction reset cleanup * Neutralize Calico ownership metadata --------- Co-authored-by: Munim Thahmid <95485323+munimthahmid@users.noreply.github.com> Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Fail alert oracle on Prometheus query errors (#912) * Decouple failure configmaps from mongo pods in `hotel-reservation` (#899) * decouple failure configmaps from mongo pods * remove empty failure configmaps creation * Support local LLM endpoints for Stratus and OpenCode (#917) * Support local LLM endpoints for Stratus * Prefer env vars for local LLM endpoints * Support local LLM endpoints for OpenCode * Route local endpoints into agent containers * Normalize OpenCode local judge models --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Preflight check the judge model (#924) * Add support for uniform agent trajectory (ATIF format) (#918) * add support for ATIF trajectory conversion and normalization, add claudecode adapter * fix: make _finish_problem idempotent * isolate .codex mount in container runner * better temp dir cleanup * add codex adapter * add opencode adapter * add copilot adapter * add stratus adapter * fix antropic caching for stratus * add gemini adapter * store normalized ATIF trajectories in SQLite indexed by problem type * edge tests * bump litellm, tikoken * update docker lib versions * Add node_clock_drift Problem (#894) * adding clock drift problem * fixing race condition causing sidecar log delay * fixing recover_fault to reverse time sync removal applied, improve cleanup * global cleanup added + fixed restoration for node_clock_drift problem * fixing mitigation oracle to ensure correct node checked * adding problem to non emulated cluster problems list * extending wait * indentation error fix * fixing time restoration in global cleanup * adding in check to make sure it picks worker node * adding another control plane filter, fixing clock step-back in global recovery * increase sidecar rollout timeout, make podnames/labels generic, idempotancy check * formatting fixes * format fix * fix * fix ruff check/format issues --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> * Minimum viable python version * Changed diagnosis oracle * Cleaned up fault recovery * Edit tasklist.yml * Removed old Java artifacts * Updated mitigation oracle to be more robust * Added polling * Fail Kafka producer leak injection on timeout --------- Co-authored-by: Saad Mohammad Rafid Pial <saadmrp222@gmail.com> Co-authored-by: Kerem Akcalioglu <blldr001@gmail.com> Co-authored-by: ermias mulugeta <61350760+ermias19@users.noreply.github.com> Co-authored-by: Tejas Shukla <85878274+TejasShukla2007@users.noreply.github.com> Co-authored-by: Petr Myagkov <73434154+Ipetr14@users.noreply.github.com> Co-authored-by: Omar Faruqe Riyad <riyad.omf@gmail.com> Co-authored-by: Mohammad Tamimul Ehsan <54908501+TamimEhsan@users.noreply.github.com> Co-authored-by: Munim Thahmid <95485323+munimthahmid@users.noreply.github.com> Co-authored-by: Om Kumar <128745874+W-OK-E@users.noreply.github.com> Co-authored-by: Munim Thahmid <munimthahmid2@gmail.com> Co-authored-by: Tanzim Hossain Romel <romel.rcs@gmail.com> Co-authored-by: haani-maybe <171415359+haani-maybe@users.noreply.github.com>
2 participants
The real-world failure story
The motivating failure case for this implementation involves a mutating admission webhook silently rewriting pod memory limits at admission time, applying a value far higher than what was expected and breaking deployments via namespace quota rejection. The Deployment spec was untouched and showed the configured limit; the wrong value only existed on the mutated pod and matched nothing in the cluster's declared configuration.
The same mechanism — a webhook silently rewriting pod resource limits at admission time — can fail in either direction. The incident above showed it with a value too high; this problem simulates the inverse, where the injected value is too low and the pod OOMKills on startup. A low limit was chosen because reliably reproducing the quota-rejection symptom depends on the cluster having tight quotas, which isn't guaranteed across environments, whereas an aggressively low limit deterministically triggers OOMKill on any reasonable workload.
Reference:
kubernetes/autoscaler#8401
How the failure is simulated in SREGym
Problem class:
sregym/conductor/problems/mutating_webhook_resource_limits.pyRegistry key:
mutating_webhook_resource_limits_social_networkThe fault builds a fully self-contained webhook backend in
platform-ops— apython:3.12-alpinecontainer running a small HTTPS server whose code lives in a ConfigMap, so no custom image or registry is needed. Fresh TLS material is generated on every injection: one CA, one server cert signed by that CA, and the correctcaBundlewired into theMutatingWebhookConfigurationso TLS validates and admission always goes through.Once the backend is up, a
MutatingWebhookConfigurationnamedpod-policy.platform.k8s.iois created scoped tosocial-networkvianamespaceSelector, withfailurePolicy: Failand a/mutatehandler. From that point every pod CREATE in the namespace gets a JSON patch injecting16Mias the first container's memory limit. Onenginx-thriftpod is then deleted to trigger an immediate mutated replacement and make the symptom visible right away.The patch logic covers three cases depending on what the incoming pod spec has — no resources field, resources but no limits, or limits already present — using the
addop throughout to avoid JSON Patch path-not-found errors.INJECTED_MEMORY_LIMIT = "16Mi"is a class constant so the injected value is defined once and flows through the webhook server code, the root cause description, and all print statements automatically.Credits:
The overall structure of the problem class and the fault injection pattern follow the approach established by @mohamedharake in
admission_webhook_tls_mismatch(#777).Runtime behaviour
Tested via the SREGym CLI on a local Kind cluster.
social-networkdeploys normally before fault injectionplatform-ops/platform-policy-controlleris running as a reachable HTTPS servicenginx-thriftpod is deleted; replacement is created and immediately mutatednginx-thriftpod repeatedly entersOOMKilled/CrashLoopBackOffkubectl get deployment nginx-thrift -o yamlshows no memory limit setkubectl describe pod <nginx-thrift-xxx>showsLimits: memory: 16Misocial-networkvianamespaceSelector; rest of cluster unaffectedMutatingWebhookConfigurationand restarting the deployment restores normal behaviourChecking the Deployment spec returns nothing for resources, but inspecting the actual running pod shows
16Miinjected under limits. The spec and the pod disagree — which only happens if something modified the pod between spec and runtime.Expected agent behaviour
Tested end-to-end with the
claudecodeagent (claude-sonnet-4-6). Diagnosis passed (composite 0.78), mitigation passed. TTL 97.6s, TTM 167.4s.Expected successful agent path:
nginx-thriftis repeatedly OOMKilling.kubectl describe podon the crashing pod and findLimits: memory: 16Mi.kubectl get mutatingwebhookconfigurationsand findpod-policy.platform.k8s.io.social-network.The intended challenge is that the visible symptom looks like an application problem — OOMKill, CrashLoopBackOff — but the root cause is a cluster-scoped admission webhook that is invisible from those angles and only discoverable by comparing what the Deployment spec declares against what the pod actually runs with.
/validate-problem mutating_webhook_resource_limits_social_network