Add Cloudflare-Style WAF Regex CPU Saturation - #773
Conversation
|
Thanks for the PR! I will review it as soon as I can. |
Saadmrp1038
left a comment
There was a problem hiding this comment.
@RedwanulKarim612 The PR looks great. I ran and tested the problem multiple times, everything works correctly.
Congrats on your first PR merged here 🎉
On a side note, you mentioned on Slack that the product-catalog startup restarts can distract the diagnosis agent from the actual fault. Wanted to provide a bit more details on that. This happens with the default barebones stratus prompt because the agent fixates on the first noisy signal it sees (restart count). With a slightly more detailed prompt that tells the agent to check pod status, resource usage, dependency graph, and events first, then merge those signals before making a decision, it looks past the noise just fine.
I ran the agent with this slightly tweaked prompt and it detected frontend-proxy pinned at its CPU limit, inspected the deployment, identified the Perl script and malicious regex, and rolled back successfully. Both diagnosis (score: 89/100) and mitigation pass.
|
/validate-problem skip |
Summary
This PR adds a new Astronomy Shop benchmark problem:
edge_request_filter_cpu_saturation.The scenario is inspired by the Cloudflare outage on July 2, 2019, where a WAF rule containing a catastrophic-backtracking regular expression caused CPU exhaustion across Cloudflare’s edge network. This benchmark recreates the same failure mode in a controlled SREGym environment: a bad edge request-filter regex is introduced into
frontend-proxy, and crafted near-matching requests drive CPU saturation and high request latency.Reference: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/
Code Changes
This PR:
EdgeRequestFilterCPUSaturation.edge_request_filter_cpu_saturation.frontend-proxyduring injection so it runs a small Perl edge request filter on port8080.load-generatorpod.Problem List.mdsregym/conductor/problems/registry.pytests/e2e-testing-scripts/registry.txtThe fault injection will:
frontend-proxydeployment.load-generatorpod.wafquery value:Expected System Behavior
frontend-proxyshould become CPU-saturated.Check pod CPU usage:
Expected signal:
Check the
frontend-proxylogs:Expected log entries:
{"event":"request_filter_eval","rule":"^([a-zA-Z]+)*$","candidateLength":5001,"elapsedSeconds":2}This means the edge request filter is spending seconds evaluating a single crafted regex input.
Mitigation
Valid mitigations include disabling the vulnerable rule, replacing the regex, or rolling back
frontend-proxy.Option 1: Disable the WAF rule
kubectl set env -n astronomy-shop deployment/frontend-proxy WAF_RULE_ENABLED=falseOption 2: Replace the vulnerable regex with a safe linear equivalent
Option 3: Roll back frontend-proxy
Wait for the rollout: