Skip to content

Latest commit

 

History

History
 
 

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

The load-test workflow

One Render Workflows project holding every task the harness drives. Deploy it once per load-test account and the runner can drive any scenario against that pool — there is nothing to choose at provisioning time.

main.py           the Workflows app; registers every task
workload.py       the fibonacci burn / sleep / ballast, shared
cpu_sleep.py      cpu_sleep
ez.py             ez, calculate_square, sum_squares
fanout.py         fanout
payload.py        payload
requirements.txt  render_sdk, pinned
Task Signature What it is for
ez total_ms Sleep and return. The latency probe: no CPU, so its end-to-end time is almost entirely queue wait and container start.
calculate_square a Arithmetic leaf.
sum_squares a, b Arithmetic root — two calculate_square children, so one dispatch is three task runs.
cpu_sleep total_ms, duty_cycle, slice_ms, fib_target_ms, ballast_mb The capacity workload. Burn interleaved with sleep, which is what makes placement measurable.
payload payload_bytes, sleep_ms, payload The object-storage round trip and the input/output size accounting.
fanout count, period_ms, leaf_ms, duty_cycle, … Root that dispatches count cpu_sleep subtasks. How one account exceeds 100 tasks/min.

calculate_square and sum_squares take the upstream examples repo's parameter names on purpose, so a pool built from this project runs hello and standard both.

Deploying it

The accounts have no GitHub connection, so the repo has to be public — a private one cannot be cloned and every build fails. Push this directory to a public repo, then point accounts create at it:

go run . accounts create -count 100 -prefix lt \
  -repo https://github.com/lytefast/render-workflows-examples-python \
  -branch loadtest -root-dir loadtest

-root-dir is the path within that repo, so the directory name there does not have to match this one. Build command pip install -r requirements.txt, start command python main.py — both are what accounts create sets.

To run it locally instead:

render workflows dev -- python main.py

Registration is in one place, and it rebinds

main.py owns the Workflows app and registers every task; the modules hold plain functions. That avoids an import cycle — no module has to import main to reach the app — but it means a subtlety is load-bearing:

task = app.task(**options)(getattr(module, name))
setattr(module, name, task)     # <- this line

A task that dispatches a subtask has to call the registered object. Calling the plain function just runs it inline in the parent's container and no subtask is ever created, which looks like a working run producing a third of the task volume it should. Putting the decorated object back under its original name means both sum_squarescalculate_square (same module) and fanoutcpu_sleep.cpu_sleep (across two) resolve to the task at call time.

Adding a task means writing the function in a module and adding one register line. Nothing else.

Why the workload isn't just sleep

cpu_sleep alternates a fibonacci burn with a sleep in slice_ms chunks. That is not incidental. The scheduler derives its packing request from observed usage (taskusagestats: mean + 1σ, clamped to [0.1, 1.0] cores), so a pure-sleep task decays to the 0.1-core floor within a few hundred runs, packing density runs ~5× past anything prod would see, and the node autoscaler never engages — the test would silently stop measuring placement.

duty_cycle is the dial:

duty_cycle Learned CPU request Use
0.0 → 0.1 core floor Control-plane max: cheapest way to push task rate
0.5 ≈ 0.5–0.6 cores Default — matches the prod default request
1.0 → 1.0 core cap Worst-case packing pressure

None of it reaches the placement decision unless workflows_usage_based_packing is enabled for the namespace. With it off the scheduler sends request = limit — the starter plan's 0.5 CPU — and the duty cycle changes nothing about packing. See config/scenarios/README.md.

fib(n) is calibrated per process against the node's actual CPU rather than hardcoded: a fixed n yields a different duty cycle on every CPU, drifting the learned request between runs and making results incomparable. The calibration target must stay well under the burn slice, because burn only checks its deadline between fib calls — a 50ms target against a 50ms slice overshoots duty 0.5 to ≈0.6. The defaults (5ms into 50ms) land within ~10%.

Keep fan-out roots bounded

Two things bound count, and both bite quietly.

Cancellation. On every task failure the scheduler runs CancelSubtasks (dbqueue.go:169): it loads the entire root tree 500 rows at a time, walks it, and cancels descendants one by one — with a 10k-row cap past which it logs hit max row cap; some task runs may be missing and stops. root_run_id propagates the whole way down, so a root that keeps spawning accumulates every descendant it ever had. At saturation, when failures are most frequent, each failure drags that whole tree through the database and cancellation quietly goes incomplete. This is worth fixing in the scheduler independently — any customer with a long-lived fan-out root hits it.

The per-owner concurrency budget, which is 20 running tasks for an unpaid team. count children of leaf_ms each, paced across period_ms, sit at count × leaf_ms / period_ms concurrent; over the budget they queue behind the limiter and the run measures the limiter rather than the cluster.

One bounded root per minute keeps each tree at a few dozen rows. standard uses 24 children of 10s across 60s — four concurrent, well inside both.

Depth does not buy headroom, only structure: the subtask counter is keyed workflow:subtask:{ownerEntityID} and does not include the parent, so a nested tree hits exactly the same 1000/min wall. If a run plateaus near that number, check taskscheduler_subtask_rate_limit_hits_total before drawing any conclusion about the scheduler.