Add node conntrack exhaustion hotel reservation problem - #768
Conversation
|
Thanks for the PR! I will review it soon. |
|
/validate-problem node_conntrack_exhaustion_hotel_reservation |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
🧪 Problem Validation —
|
| Stage | Status | Detail |
|---|---|---|
| Resolve problem in registry | ✅ | NodeConntrackExhaustionHotelReservation · app Hotel Reservation |
| Deploy application | ✅ | Hotel Reservation deployed to namespace hotel-reservation |
| Inject fault | ✅ | inject_fault() completed without error |
| Oracle fails after fault injection | ✅ | oracle reported failure after 1 check(s) |
| Recover fault | ✅ | recover_fault() completed without error |
| Oracle passes after recovery | ✅ | oracle reported success after 1 check(s) |
Lifecycle: deploy app → inject fault → oracle fails → recover fault → oracle passes.
Workflow run · commit 7a511e27
There was a problem hiding this comment.
The kind crash happened because my user was not added to the docker group and I just used sudo docker for any docker related work which is not common. The current docker exec approch should be fine.
Please take a look at the other crash. I marked the possible reason in the comment below.
|
@Saadmrp1038 I pushed the latest changes. I addressed the review points by bounding nf_conntrack_max during injection and restoring the original value during recovery, replacing the many client replicas with a single multiprocessing client plus multi-port gateway, pinning both to the same victim node, keeping the kind docker exec path with a portable hostNetwork fallback, using generic helper pod names, and relaxing mitigation scoring so valid fixes are accepted based on conntrack recovery plus frontend health. |
Saadmrp1038
left a comment
Saadmrp1038
left a comment
There was a problem hiding this comment.
@munimthahmid some minor issues found but other than those everything looks okay. I will merge the PR after you address them.
On a side note, I did some agent runs on this fault. Claude code can detect something fishy is going on but can't actually pinpoint the fault (I removed the print statement while testing). It notices abnormal behaviour like kubectl exec failing on the victim node and realizes there's something wrong with the clulster's networking. But it gets stuck in debuggin spiral and times out.
| return { | ||
| "name": "gateway", | ||
| "image": "python:3.12-alpine", | ||
| "command": ["python", "-c", script], |
There was a problem hiding this comment.
The default container soft ulimit is 1024 file descriptors on most clusters (in my case CloudLab Cluster). The client needs ~275,000 sockets. I got the following error:
OSError: [Errno 24] No file descriptors available
Passing the script via env var and raising ulimit before exec worked for me:
"command": ["sh", "-c", "ulimit -n 524288; exec python -c \"$SCRIPT\""],
"env": [
...,
{"name": "SCRIPT", "value": script},
],
| return { | ||
| "name": "client", | ||
| "image": "python:3.12-alpine", | ||
| "command": ["python", "-c", script], |
There was a problem hiding this comment.
Same ulimit issue as the client. The gateway needs to accept() and hold ~275k sockets.
| while all(worker.is_alive() for worker in workers): | ||
| count = open("/host-proc/sys/net/netfilter/nf_conntrack_count").read().strip() | ||
| maximum = open("/host-proc/sys/net/netfilter/nf_conntrack_max").read().strip() | ||
| print(f"workers={len(workers)} target={total} nf_conntrack_count={count} nf_conntrack_max={maximum}", flush=True) |
There was a problem hiding this comment.
An agent checking logs will immediately see these print statements. The nf_conntrack_count and nf_conntrack_max in the logs will leak the fault mechanism to the agent. We should either remove this print or not say conntrack here.
|
/validate-problem node_conntrack_exhaustion_hotel_reservation |
bf9a1da to
8d36c3b
Compare
|
@Saadmrp1038 I addressed the minor issues and pushed the update. Both the gateway and client now raise ulimit -n before execing Python,the scripts are passed through env vars, and I removed the conntrack-specific client log output plus the unused /host-proc mount. On the side note, that matches what I saw as well. The fault was already subtle, and now that the explicit conntrack log hint is removed, it becomes even harder to diagnose. In most of my runs, the agent spent a lot of time in diagnosis and timed out during mitigation; after increasing the timeout by 10 minutes, it was able to mitigate, although that run still failed diagnosis with 67/100. Do you think we should tune anything to make the fault more diagnosable? |
|
@munimthahmid Great work!
Let's keep it as it is now. Since the fault mechanism itself is working correctly, we can think about what the agent would need to do later. This is a really hard problem for the agent, since the agent can't really exec/SSH into the victim node (as conntrack is full) and run diagnostics on the victim node directly. |
hard is good :)) Great work @munimthahmid !! Thank you for working hard towards the end! |
4 participants
This PR adds a new SREGym problem that reproduces a production Kubernetes failure class where the visible symptoms look like application, DNS, or observability flakiness, but the real fault is node-level Linux connection-tracking exhaustion.
The new benchmark problem is:
1. The Real-World Failure Story
This problem is inspired by Mark Betz's production incident report, "Exhausting conntrack table space crippled our k8s cluster", also mirrored on Medium.
In that incident, the first symptoms looked like ordinary service/network trouble: slow RPC services, intermittent failures, and DNS-looking errors such as temporary name-resolution failures. The actual root cause was lower-level. A small number of Kubernetes nodes had accumulated enough active network flows to fill the Linux
nf_conntracktable. Once that table was full, new connection attempts could be dropped even though pods, services, endpoints, and the control plane could still look mostly normal.The interesting operational lesson is that this is not primarily a "pod crashed" failure. It is a node networking state-exhaustion failure. Existing connections may continue, many Kubernetes resources still look healthy, and the misleading symptoms can send responders toward CoreDNS, Jaeger, Consul, databases, or random application logs before they inspect the node's
nf_conntrack_countandnf_conntrack_max.This PR reproduces that failure mode in SREGym: a connection-heavy workload is concentrated on one worker node until that node's conntrack table is exhausted, causing fresh Hotel Reservation traffic from that node to time out while the application objects still appear mostly healthy.
2. How the Failure Is Simulated
The new problem lives in:
Fault injection creates two support workloads inside the
hotel-reservationnamespace:rpc-gatewayis a small Python TCP listener. It accepts connections and holds them open.edge-traffic-clientis pinned to the victim worker node. Each client pod opens many TCP connections torpc-gatewayand keeps them open. This fills the victim node's Linux conntrack table, not a Kubernetes Service endpoint table.The injection is calibrated against the victim node's actual conntrack capacity:
The current implementation reads
nf_conntrack_maxat injection time and computes enough client replicas to target roughly 105% of the table capacity, capped bymax_client_replicasso an unexpectedly large local cluster fails early instead of creating an unbounded workload.On the local kind cluster used for validation,
nf_conntrack_maxwas262144, so the injected workload is approximately:That scale is large enough to saturate the node conntrack table while still keeping the benchmark contained to the local kind cluster.
3. Runtime Behavior
After fault injection, the failure is intentionally non-obvious:
These can look mostly healthy. The failure is not that the frontend Service has no endpoints, not that CoreDNS is deleted, and not that Jaeger is broken.
The deterministic broken-state signal is on the victim node:
Useful debugging commands:
Expected evidence in the broken state:
The failure should be diagnosed as node-level conntrack exhaustion caused by
edge-traffic-clientopening excessive held TCP connections torpc-gateway.4. Expected Diagnosis
A correct diagnosis should identify:
Common incorrect diagnoses include:
Those are symptoms or distractions, not the root cause.
5. Expected Mitigation
A correct mitigation should reduce the connection pressure and verify that the victim node recovers.
Valid fixes include:
The mitigation oracle is implemented in:
It intentionally goes beyond the generic "all pods are Running" check. That generic check is insufficient for this failure because pods can be Running while the node conntrack table remains full.
The custom oracle verifies:
This catches symptom-only fixes where the application objects look healthy but node networking is still saturated.
6. Validation
Local validation was run repeatedly on a 4-node kind cluster.
Clean scored results:
89/1000.29%.100/1000.27%.100/10089/10011/100Stratus difficulty signal:
Across seven scored Stratus GPT-5 attempts, diagnosis did not pass. One attempt still recovered the cluster by deleting the synthetic workloads, but the submitted diagnosis missed the node-level conntrack mechanism. Six scored attempts produced failed diagnoses and then timed out before a successful mitigation result. Common incorrect diagnoses included Jaeger/tracing misconfiguration, Consul startup noise, CoreDNS degradation, profile database connectivity, and generic service bootstrap problems.
This is consistent with the intended difficulty: the visible symptoms are misleading unless the agent connects workload placement and TCP connection fanout to host-level
nf_conntrackstate.Several additional local Stratus runs were excluded because LiteLLM/OpenAI retry exhaustion occurred before the agent produced a meaningful scored result. These are disclosed for completeness, but they are not counted as problem failures because the agent did not receive a completed model response.
No new Python dependencies are added.
No agent RBAC expansion is required.
Fault recovery is idempotent: if an agent already deletes or scales down the synthetic support resources,
recover_fault()tolerates the missing resources and waits for conntrack usage to drop.