Conversation
Add a self-contained perf-regression job to the nightly that runs a
redis-benchmark suite against the freshly built module and compares
throughput (ops/sec) against the previous nightly's baseline artifact.
- A metric >5% slower than the baseline fails the nightly (turns it red).
- The baseline is a ratchet: it only moves in the faster direction
automatically. On a regression the old (faster) baseline is kept so the
bar does not erode silently.
- To deliberately accept a slower baseline ("lower the bar"), re-run the
nightly via workflow_dispatch with approve-lower-baseline=true.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RabQEggYzz3s3JJ16ZUJ3i
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1626 +/- ##
=======================================
Coverage 85.97% 85.97%
=======================================
Files 15 15
Lines 5305 5305
=======================================
Hits 4561 4561
Misses 744 744 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1795726. Configure here.
Replaces the runner-based benchmark from the previous commit. GitHub runners are not a trustworthy source of performance signal, and the EC2 perf suite already pushes every result to RedisTimeSeries, so the gate now reads those series back instead of measuring anything itself. perf_regression_gate.py queries the same labels redisbench-admin's own `compare` uses (branch / metric / test_name / deployment_name / triggering_env / github_org / github_repo) and judges the newest results against a ratcheted baseline: - The baseline is the best trailing-window median in the query window, so it only ever moves in the faster direction -- a slow run cannot lower it and performance cannot erode silently. - Ratcheting on a median-of-N rather than on single runs is what makes that usable: EC2 variance is large, and a max over individual samples would latch onto an outlier and then fail forever. - The threshold widens to the noise floor measured for each test; past --max-threshold-pct observed noise a test is reported as not gateable rather than flapping. - An empty result set is a hard failure. `compare` itself never fails a build, and a gate that silently matches no series must not be mistaken for one that passed. Lowering the bar is a reviewed change to tests/benchmarks/perf-baselines.json, recording the value, who approved it and why -- nothing the gate does can lower it on its own. Wired after each merge to master (attributing a regression to its commit) and nightly as a safety net. Both start with enforce: false so the thresholds can be calibrated against real EC2 variance before the gate turns anything red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Summary
Adds a day-over-day performance regression gate to the nightly (
event-nightly.yml) via a new reusable workflow,flow-perf-regression.yml. It mirrors the shape of the existing self-contained memory-regression ratchet in this repo, but for throughput (ops/sec).How it works
redis-benchmarksuite against the freshly built module (no AWS / RedisTimeSeries needed) and records ops/sec per command — best-of-3 per command to damp CI noise.perf-report-baselinefromevent-nightly.ymlonmaster) and compares each metric.approve-lower-baseline = trueto deliberately accept a slower baseline. Scheduled nightlies always passfalse, so the baseline can never auto-lower — the only way down is a human-triggered dispatch (Actions/write permission is the approval gate).Benchmarked commands
JSON.GET(scalar / full-doc / nested),JSON.SET,JSON.NUMINCRBY,JSON.ARRLEN— seeded against a small doc and a ~500-element array doc.Changes
.github/workflows/flow-perf-regression.yml— new reusable workflow: benchmark → compare → ratchet → gate → publish baseline artifact + detailed report..github/workflows/event-nightly.yml— newperf-regressionjob wired into the nightly, anapprove-lower-baselineworkflow_dispatchinput, and the job added to thetest-summaryfailure list.Where results show up
The per-metric table (Baseline / Today / Change / New baseline / Status) is written to the job's GitHub run summary page, alongside the existing memory (RAM overhead) report — no Slack changes.
Notes / follow-ups
benchmark-flow.yml; the best-of-3 + ratchet design mitigates false positives, and the threshold is a single workflow input if it needs tuning. If preferred, this can instead gate on the EC2 numbers already flowing into RedisTimeSeries/Grafana.🤖 Generated with Claude Code
Note
Medium Risk
New CI depends on perf RedisTimeSeries secrets and query filters; misconfiguration fails the gate (exit 2), but enforcement is off until thresholds are calibrated.
Overview
Adds a performance regression gate that reads throughput metrics the EC2 benchmark suite already stores in RedisTimeSeries—no new benchmarks on GitHub runners. A new reusable workflow
flow-perf-regression.ymlrunsperf_regression_gate.py(Python +PERFORMANCE_RTS_*secrets), writes a job summary and JSON artifact, and supports optional--enforce(currently warn-only everywhere).Baseline logic: for each test/metric, compare the median of the newest N runs against a ratcheting baseline (best trailing-window median in history, excluding the current window). Thresholds widen with observed noise; very noisy tests are skipped. Lowering the bar requires a reviewed entry in
perf-baselines.json(tracked despite*.jsonin.gitignore).CI wiring: after merge benchmarks in
event-push-to-integ.yml(needs: benchmark), and as a standalone nightly safety net inevent-nightly.yml(included intest-summaryfailures). Docs and unit tests cover operation, calibration before turning on enforcement, and the decision logic without Redis.Reviewed by Cursor Bugbot for commit 9c508b9. Bugbot is set up for automated code reviews on this repo. Configure here.