One Render Workflows project holding every task the harness drives. Deploy it once per load-test account and the runner can drive any scenario against that pool — there is nothing to choose at provisioning time.
main.py the Workflows app; registers every task
workload.py the fibonacci burn / sleep / ballast, shared
cpu_sleep.py cpu_sleep
ez.py ez, calculate_square, sum_squares
fanout.py fanout
payload.py payload
requirements.txt render_sdk, pinned
| Task | Signature | What it is for |
|---|---|---|
ez |
total_ms |
Sleep and return. The latency probe: no CPU, so its end-to-end time is almost entirely queue wait and container start. |
calculate_square |
a |
Arithmetic leaf. |
sum_squares |
a, b |
Arithmetic root — two calculate_square children, so one dispatch is three task runs. |
cpu_sleep |
total_ms, duty_cycle, slice_ms, fib_target_ms, ballast_mb |
The capacity workload. Burn interleaved with sleep, which is what makes placement measurable. |
payload |
payload_bytes, sleep_ms, payload |
The object-storage round trip and the input/output size accounting. |
fanout |
count, period_ms, leaf_ms, duty_cycle, … |
Root that dispatches count cpu_sleep subtasks. How one account exceeds 100 tasks/min. |
calculate_square and sum_squares take the upstream examples repo's
parameter names on purpose, so a pool built from this project runs
hello and
standard both.
The accounts have no GitHub connection, so the repo has to be public — a
private one cannot be cloned and every build fails. Push this directory to a
public repo, then point accounts create at it:
go run . accounts create -count 100 -prefix lt \
-repo https://github.com/lytefast/render-workflows-examples-python \
-branch loadtest -root-dir loadtest-root-dir is the path within that repo, so the directory name there does not
have to match this one. Build command pip install -r requirements.txt, start
command python main.py — both are what accounts create sets.
To run it locally instead:
render workflows dev -- python main.pymain.py owns the Workflows app and registers every task; the modules hold
plain functions. That avoids an import cycle — no module has to import main to
reach the app — but it means a subtlety is load-bearing:
task = app.task(**options)(getattr(module, name))
setattr(module, name, task) # <- this lineA task that dispatches a subtask has to call the registered object. Calling
the plain function just runs it inline in the parent's container and no subtask
is ever created, which looks like a working run producing a third of the task
volume it should. Putting the decorated object back under its original name
means both sum_squares → calculate_square (same module) and fanout →
cpu_sleep.cpu_sleep (across two) resolve to the task at call time.
Adding a task means writing the function in a module and adding one register
line. Nothing else.
cpu_sleep alternates a fibonacci burn with a sleep in slice_ms chunks. That
is not incidental. The scheduler derives its packing request from observed usage
(taskusagestats: mean + 1σ, clamped to [0.1, 1.0] cores), so a pure-sleep
task decays to the 0.1-core floor within a few hundred runs, packing density runs
~5× past anything prod would see, and the node autoscaler never engages — the
test would silently stop measuring placement.
duty_cycle is the dial:
duty_cycle |
Learned CPU request | Use |
|---|---|---|
| 0.0 | → 0.1 core floor | Control-plane max: cheapest way to push task rate |
| 0.5 | ≈ 0.5–0.6 cores | Default — matches the prod default request |
| 1.0 | → 1.0 core cap | Worst-case packing pressure |
None of it reaches the placement decision unless workflows_usage_based_packing
is enabled for the namespace. With it off the scheduler sends request = limit —
the starter plan's 0.5 CPU — and the duty cycle changes nothing about packing.
See config/scenarios/README.md.
fib(n) is calibrated per process against the node's actual CPU rather than
hardcoded: a fixed n yields a different duty cycle on every CPU, drifting the
learned request between runs and making results incomparable. The calibration
target must stay well under the burn slice, because burn only checks its
deadline between fib calls — a 50ms target against a 50ms slice overshoots duty
0.5 to ≈0.6. The defaults (5ms into 50ms) land within ~10%.
Two things bound count, and both bite quietly.
Cancellation. On every task failure the scheduler runs CancelSubtasks
(dbqueue.go:169): it loads the entire root tree
500 rows at a time, walks it, and cancels descendants one by one — with a 10k-row
cap past which it logs hit max row cap; some task runs may be missing and
stops. root_run_id propagates the whole way down, so a root that keeps spawning
accumulates every descendant it ever had. At saturation, when failures are most
frequent, each failure drags that whole tree through the database and
cancellation quietly goes incomplete. This is worth fixing in the scheduler
independently — any customer with a long-lived fan-out root hits it.
The per-owner concurrency budget, which is 20 running tasks for an unpaid
team. count children of leaf_ms each, paced across period_ms, sit at
count × leaf_ms / period_ms concurrent; over the budget they queue behind the
limiter and the run measures the limiter rather than the cluster.
One bounded root per minute keeps each tree at a few dozen rows. standard uses
24 children of 10s across 60s — four concurrent, well inside both.
Depth does not buy headroom, only structure: the subtask counter is keyed
workflow:subtask:{ownerEntityID} and does not include the parent, so a nested
tree hits exactly the same 1000/min wall. If a run plateaus near that number,
check taskscheduler_subtask_rate_limit_hits_total before drawing any conclusion
about the scheduler.