Add concurrency sweep in presets by Bihan · Pull Request #4470 · dstackai/dstack

Bihan · 2026-11-05T10:59:46Z

Add concurrency sweeps to presets
Preset creation can now benchmark a service at multiple concurrency levels in each trial.

Configuration
The concurrency field accepts a single value, a list, comma-separated values, or a doubling range:

concurrency: 4
concurrency: [2, 4, 8]
concurrency: 2,4,8
concurrency: 2..8

Values are normalized into a sorted list of unique positive integers.

CLI output
dstack preset list -a shows the requested sweep and benchmark ranges. The c= value in the BENCHMARK column identifies the selected point:

NAME                           ID        CONSTRAINTS                                        BENCHMARK                                            STATUS
muse-glimmer-nvfp4-agentx-v2   f07f995b  data=semianalysisai/cc-traces-weka-062126 c=2..32  tps/user=14..92.7 c=16 ttft=360ms..610ms ctx=128K  verified (2/2)

The sparkline continues to show one throughput value per trial. For each trial, the displayed value comes from the selected sweep point.

Selection Rule
For each trial, the CLI selects the benchmark point as follows:

Consider points whose p50 TTFT meets max_ttft.
Choose the point with the highest output throughput.
If no point meets max_ttft, choose the point with the lowest p50 TTFT. If TTFT values tie, choose the higher throughput.

For a completed preset, dstack preset get --json <preset-id> includes all final benchmarks and the selected benchmark:

{
  "id": "<preset-id>",
  "benchmarks": [
    { "workload": { "concurrency": 4 }, "metrics": { "...": "..." } },
    { "workload": { "concurrency": 8 }, "metrics": { "...": "..." } },
    { "workload": { "concurrency": 16 }, "metrics": { "...": "..." } }
  ],
  "benchmark": {
    "workload": { "concurrency": 16 },
    "metrics": { "...": "..." }
  }
}

Compatibility

Older saved presets containing a single benchmark remain readable: the benchmark is treated as a one-point sweep.
When an older session is used as a previous reference, its records are converted to the sweep format in the new session’s workspace. The original records remain unchanged.
Older successful agent reports containing a single benchmark are accepted and converted to a one-item list before verification.

Update system_prompt.md Update system_prompt.md Update system_prompt.md Update system_prompt.md Update system_prompt.md

Update system_prompt.md

peterschmidt85 · 2026-11-12T10:19:55Z

+one `output_tok_per_s` value for each concurrency. Before comparing trials,
+decide on a method to compare their `output_tok_per_s` values across
+concurrencies, and keep that method for the whole session. Report the chosen


What exactly we want to say with "Before comparing trials,
decide on a method to compare their output_tok_per_s values across
concurrencies, and keep that method for the whole session."? I have a feeling I understand what you mean but 1) could be more clear; 2) again, not sure if Constraints is the right place for this - see my other comment below.

To be more clear maybe we provide example scenario.

For example Trial 1 might have [400, 800, 850] output_tok_per_s for concurrencies [16, 32, 64] and Trail 2 might have [800, 1200, null] output_tok_per_s for concurrencies [16, 32, 64].

Agent has to decide whether

method1
it will choose Trial 1 because benchmark was successful at all concurrencies.
OR
method2
it will choose Trial 2 because benchmark at concurrency gave highest output_tok_per_s.

Agent has to decide method1 or method2 and why it choose before comparing trials.

peterschmidt85 · 2026-11-12T10:21:10Z

+Each trial is benchmarked at every value of `concurrency`, so a trial can have
+one `output_tok_per_s` value for each concurrency. Before comparing trials,
+decide on a method to compare their `output_tok_per_s` values across
+concurrencies, and keep that method for the whole session. Report the chosen
+method via `progress` (see `# Progress`).<!--?if prompt--> If `## Additional instructions`
+says how to compare trials, use that method.<!--?end-->


Also why "Each trial is benchmarked at every value of concurrency, so a trial can have
one output_tok_per_s value for each concurrency. Before comparing trials,
decide on a method to compare their output_tok_per_s values across
concurrencies, and keep that method for the whole session. Report the chosen
method via progress (see # Progress). If ## Additional instructions
says how to compare trials, use that method."

is in Constraints? It talks about trials / how to select a trial.

For example, we have "The end goal is to find a model serving configuration that matches the constraints and delivers the best performance." as well as "A trial is also failed when its benchmark at all concurrency values fails to meet the constraints (see # Constraints)." where talking on how constraints are applied to trials and that seems to be better places - not in Constraints.

The idea is to minimize the contract of each section to make explicit the role and responsibility of each section as clear and explicit as possible.

I choose # Constraints for 2 reason.

I want it to be just before we talk about "best performance".

It is kept before the paragraph

During the trials and experimentation aimed at the best performance, you may
pick the hardware (the best available within the allowed dstack fleets)...

We define fields: trials and concurrency required to explain about the comparison in constraints section.

peterschmidt85 · 2026-11-12T10:30:27Z

-`final_report.json.benchmark` (values are illustrative):
+Record every benchmark using the following structure and field names (values
+are illustrative): for a trial, in `trials/<n>/trial.json` as
+`results[concurrency == <c>].benchmark`; for the final service, as


(Minor) I assumed we define the format in the other place - trials/<n>/trial.json is a JSON object. Here we just say where to save it. Perhaps we could just make explicit again that benchmark result is per concurrency value.

peterschmidt85 · 2026-11-12T10:31:05Z


 ```
-{"resources": {...}, "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...}
+{"resources": {...}, "results": [{"concurrency": ..., "benchmark": {...}, "context_length": ..., "learned": ..., "failed": ...}, ...]}


BTW why "results" and not "result". IMO its one result.

peterschmidt85 · 2026-11-12T10:33:49Z

-`final_report.json.benchmark.command`.
+Do not put secret values in `final_report.json`, or print them. Use env
+references in `final_report.json.service_yaml`; use environment variable
+names or redacted values in `final_report.json.results[].benchmark.command`.


Suggested change

names or redacted values in `final_report.json.results[].benchmark.command`.

names or redacted values in `final_report.json.results[concurrency == <c>].benchmark.command`.

peterschmidt85 · 2026-11-12T10:34:51Z

+  from `constraints.json`. Each entry contains:
+  - `concurrency`: the concurrency value for this result
+  - `context_length`: the largest context verified for the final service at
+    this concurrency, as described in `## Benchmark`
+  - `benchmark`: the final service benchmark at this concurrency, as
+    described in `## Benchmark`


"Each entry contains" is vauge format definition, no??

Do you mean instead we say
Each entry is an object with fields no other than mentioned below.

concurrency:

context_length:

Bihan Rana added 2 commits November 5, 2026 13:37

Add concurrency sweeps to presets

0ac666b

Minor Update

8e1efac

Bihan requested a review from peterschmidt85 November 5, 2026 10:59

Bihan Rana added 2 commits November 9, 2026 14:42

Update system_prompt.md

7533695

Update system_prompt.md Update system_prompt.md Update system_prompt.md Update system_prompt.md Update system_prompt.md

Update system_prompt.md

5995cf6

Update system_prompt.md

peterschmidt85 reviewed Nov 12, 2026

View reviewed changes

Update system_prompt.md

c952822

Navigation Menu

Repository navigation

Add concurrency sweep in presets - #4470

Add concurrency sweep in presets#4470
Bihan wants to merge 5 commits into
dstackai:masterfrom
Bihan:add_concurrency_sweep_in_presets

Bihan commented Nov 5, 2026

Uh oh!

peterschmidt85 Nov 12, 2026 •

edited

Loading

Uh oh!

Bihan Nov 12, 2026 •

edited

Loading

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

Bihan Nov 12, 2026

Uh oh!

peterschmidt85 Nov 12, 2026 •

edited

Loading

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

peterschmidt85 Nov 12, 2026

Uh oh!

Bihan Nov 12, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	names or redacted values in `final_report.json.results[].benchmark.command`.
	names or redacted values in `final_report.json.results[concurrency == <c>].benchmark.command`.

Conversation

Bihan commented Nov 5, 2026

Uh oh!

peterschmidt85 Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bihan Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Bihan Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

peterschmidt85 Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Bihan Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

peterschmidt85 Nov 12, 2026 •

edited

Loading

Bihan Nov 12, 2026 •

edited

Loading

peterschmidt85 Nov 12, 2026 •

edited

Loading

Bihan Nov 12, 2026 •

edited

Loading