Skip to content

Add Cloudflare-Style WAF Regex CPU Saturation - #773

Merged
Saadmrp1038 merged 3 commits into
SREGym:mainfrom
RedwanulKarim612:regex-cpu-saturation
May 24, 2026
Merged

Add Cloudflare-Style WAF Regex CPU Saturation#773
Saadmrp1038 merged 3 commits into
SREGym:mainfrom
RedwanulKarim612:regex-cpu-saturation

Conversation

@RedwanulKarim612

Copy link
Copy Markdown
Contributor

Summary

This PR adds a new Astronomy Shop benchmark problem: edge_request_filter_cpu_saturation.

The scenario is inspired by the Cloudflare outage on July 2, 2019, where a WAF rule containing a catastrophic-backtracking regular expression caused CPU exhaustion across Cloudflare’s edge network. This benchmark recreates the same failure mode in a controlled SREGym environment: a bad edge request-filter regex is introduced into frontend-proxy, and crafted near-matching requests drive CPU saturation and high request latency.
Reference: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/

Code Changes

This PR:

  • Adds EdgeRequestFilterCPUSaturation.
  • Registers the problem as edge_request_filter_cpu_saturation.
  • Adds a mitigation oracle for checking whether the vulnerable regex is no longer active.
  • Patches frontend-proxy during injection so it runs a small Perl edge request filter on port 8080.
  • Configures the filter with the vulnerable regex:
^([a-zA-Z]+)*$
  • Starts crafted replay traffic from the existing load-generator pod.
  • Adds the problem to:
    • Problem List.md
    • sregym/conductor/problems/registry.py
    • tests/e2e-testing-scripts/registry.txt

The fault injection will:

  1. Patch the frontend-proxy deployment.
  2. Replace the normal proxy command with a small Perl edge request filter.
  3. Enable the vulnerable WAF regex:
^([a-zA-Z]+)*$
  1. Start crafted replay traffic from the existing load-generator pod.
  2. Send repeated requests containing a long near-matching waf query value:
aaaaaaaaaaaaaaaaa...!

Expected System Behavior

frontend-proxy should become CPU-saturated.

Check pod CPU usage:

kubectl top pods -n astronomy-shop

Expected signal:

frontend-proxy   100m

Check the frontend-proxy logs:

kubectl logs -n astronomy-shop deployment/frontend-proxy --tail=100

Expected log entries:

{"event":"request_filter_eval","rule":"^([a-zA-Z]+)*$","candidateLength":5001,"elapsedSeconds":2}

This means the edge request filter is spending seconds evaluating a single crafted regex input.

Mitigation

Valid mitigations include disabling the vulnerable rule, replacing the regex, or rolling back frontend-proxy.

Option 1: Disable the WAF rule

kubectl set env -n astronomy-shop deployment/frontend-proxy WAF_RULE_ENABLED=false

Option 2: Replace the vulnerable regex with a safe linear equivalent

kubectl set env -n astronomy-shop deployment/frontend-proxy WAF_RULE_REGEX='^[A-Za-z]+$'

Option 3: Roll back frontend-proxy

kubectl rollout undo -n astronomy-shop deployment/frontend-proxy

Wait for the rollout:

kubectl rollout status -n astronomy-shop deployment/frontend-proxy --timeout=120s

@Saadmrp1038

Copy link
Copy Markdown
Collaborator

Thanks for the PR! I will review it as soon as I can.

@Saadmrp1038
Saadmrp1038 self-requested a review May 24, 2026 04:51
@Saadmrp1038 Saadmrp1038 added the problem Adding a new problem to the benchmark label May 24, 2026

@Saadmrp1038 Saadmrp1038 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RedwanulKarim612 The PR looks great. I ran and tested the problem multiple times, everything works correctly.

Congrats on your first PR merged here 🎉


On a side note, you mentioned on Slack that the product-catalog startup restarts can distract the diagnosis agent from the actual fault. Wanted to provide a bit more details on that. This happens with the default barebones stratus prompt because the agent fixates on the first noisy signal it sees (restart count). With a slightly more detailed prompt that tells the agent to check pod status, resource usage, dependency graph, and events first, then merge those signals before making a decision, it looks past the noise just fine.

I ran the agent with this slightly tweaked prompt and it detected frontend-proxy pinned at its CPU limit, inspected the deployment, identified the Perl script and malicious regex, and rolled back successfully. Both diagnosis (score: 89/100) and mitigation pass.

@Saadmrp1038

Copy link
Copy Markdown
Collaborator

/validate-problem skip

@Saadmrp1038
Saadmrp1038 merged commit 2ee8d04 into SREGym:main May 24, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

problem Adding a new problem to the benchmark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants