Skip to content

ci: nightly day-over-day performance regression gate - #1626

Open
gabsow wants to merge 2 commits into
masterfrom
claude/perf-regression-nightly-check-prdqkz
Open

gabsow wants to merge 2 commits into
masterfrom
claude/perf-regression-nightly-check-prdqkz

Conversation

@gabsow

@gabsow gabsow commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a day-over-day performance regression gate to the nightly (event-nightly.yml) via a new reusable workflow, flow-perf-regression.yml. It mirrors the shape of the existing self-contained memory-regression ratchet in this repo, but for throughput (ops/sec).

How it works

  1. Builds Redis + RedisJSON, then runs a self-contained redis-benchmark suite against the freshly built module (no AWS / RedisTimeSeries needed) and records ops/sec per command — best-of-3 per command to damp CI noise.
  2. Downloads the previous nightly's baseline artifact (perf-report-baseline from event-nightly.yml on master) and compares each metric.
  3. A metric more than 5% slower than the baseline fails the job (turns the nightly red).
  4. Ratchet: the baseline moves only in the faster direction automatically. On a regression the old (faster) baseline is kept, so the bar never erodes silently.
  5. Manual approval to lower the bar: re-run the nightly via Run workflow with approve-lower-baseline = true to deliberately accept a slower baseline. Scheduled nightlies always pass false, so the baseline can never auto-lower — the only way down is a human-triggered dispatch (Actions/write permission is the approval gate).

Benchmarked commands

JSON.GET (scalar / full-doc / nested), JSON.SET, JSON.NUMINCRBY, JSON.ARRLEN — seeded against a small doc and a ~500-element array doc.

Changes

  • .github/workflows/flow-perf-regression.yml — new reusable workflow: benchmark → compare → ratchet → gate → publish baseline artifact + detailed report.
  • .github/workflows/event-nightly.yml — new perf-regression job wired into the nightly, an approve-lower-baseline workflow_dispatch input, and the job added to the test-summary failure list.

Where results show up

The per-metric table (Baseline / Today / Change / New baseline / Status) is written to the job's GitHub run summary page, alongside the existing memory (RAM overhead) report — no Slack changes.

Notes / follow-ups

  • Throughput on shared GitHub runners is noisier than the dedicated EC2 perf infra used by benchmark-flow.yml; the best-of-3 + ratchet design mitigates false positives, and the threshold is a single workflow input if it needs tuning. If preferred, this can instead gate on the EC2 numbers already flowing into RedisTimeSeries/Grafana.
  • An analogous change for RedisBloom is prepared but is out of scope for this PR.

🤖 Generated with Claude Code


Note

Medium Risk
New CI depends on perf RedisTimeSeries secrets and query filters; misconfiguration fails the gate (exit 2), but enforcement is off until thresholds are calibrated.

Overview
Adds a performance regression gate that reads throughput metrics the EC2 benchmark suite already stores in RedisTimeSeries—no new benchmarks on GitHub runners. A new reusable workflow flow-perf-regression.yml runs perf_regression_gate.py (Python + PERFORMANCE_RTS_* secrets), writes a job summary and JSON artifact, and supports optional --enforce (currently warn-only everywhere).

Baseline logic: for each test/metric, compare the median of the newest N runs against a ratcheting baseline (best trailing-window median in history, excluding the current window). Thresholds widen with observed noise; very noisy tests are skipped. Lowering the bar requires a reviewed entry in perf-baselines.json (tracked despite *.json in .gitignore).

CI wiring: after merge benchmarks in event-push-to-integ.yml (needs: benchmark), and as a standalone nightly safety net in event-nightly.yml (included in test-summary failures). Docs and unit tests cover operation, calibration before turning on enforcement, and the decision logic without Redis.

Reviewed by Cursor Bugbot for commit 9c508b9. Bugbot is set up for automated code reviews on this repo. Configure here.

Add a self-contained perf-regression job to the nightly that runs a
redis-benchmark suite against the freshly built module and compares
throughput (ops/sec) against the previous nightly's baseline artifact.

- A metric >5% slower than the baseline fails the nightly (turns it red).
- The baseline is a ratchet: it only moves in the faster direction
  automatically. On a regression the old (faster) baseline is kept so the
  bar does not erode silently.
- To deliberately accept a slower baseline ("lower the bar"), re-run the
  nightly via workflow_dispatch with approve-lower-baseline=true.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RabQEggYzz3s3JJ16ZUJ3i
@CLAassistant

CLAassistant commented Jul 30, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ gabsow
❌ claude
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.97%. Comparing base (5ca803d) to head (9c508b9).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1626   +/-   ##
=======================================
  Coverage   85.97%   85.97%           
=======================================
  Files          15       15           
  Lines        5305     5305           
=======================================
  Hits         4561     4561           
  Misses        744      744           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1795726. Configure here.

Comment thread .github/workflows/flow-perf-regression.yml Outdated
@gabsow gabsow changed the title ci: add day-over-day performance regression gate to nightly ci: nightly day-over-day performance regression gate Jul 30, 2026
Replaces the runner-based benchmark from the previous commit. GitHub runners are not a
trustworthy source of performance signal, and the EC2 perf suite already pushes every result
to RedisTimeSeries, so the gate now reads those series back instead of measuring anything
itself.

perf_regression_gate.py queries the same labels redisbench-admin's own `compare` uses
(branch / metric / test_name / deployment_name / triggering_env / github_org / github_repo)
and judges the newest results against a ratcheted baseline:

- The baseline is the best trailing-window median in the query window, so it only ever moves
  in the faster direction -- a slow run cannot lower it and performance cannot erode silently.
- Ratcheting on a median-of-N rather than on single runs is what makes that usable: EC2
  variance is large, and a max over individual samples would latch onto an outlier and then
  fail forever.
- The threshold widens to the noise floor measured for each test; past --max-threshold-pct
  observed noise a test is reported as not gateable rather than flapping.
- An empty result set is a hard failure. `compare` itself never fails a build, and a gate that
  silently matches no series must not be mistaken for one that passed.

Lowering the bar is a reviewed change to tests/benchmarks/perf-baselines.json, recording the
value, who approved it and why -- nothing the gate does can lower it on its own.

Wired after each merge to master (attributing a regression to its commit) and nightly as a
safety net. Both start with enforce: false so the thresholds can be calibrated against real
EC2 variance before the gate turns anything red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants