Repository navigation
Conversation
更新 MiniMax-M3.1-Flash GB200 SGLang AgentX 配置。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| cuda-graph-max-bs-decode: 64 | ||
| mem-fraction-static: 0.81 | ||
| enable-hierarchical-cache: true | ||
| hicache-size: 200 |
There was a problem hiding this comment.
🔴 The new DRAM points pin about 800 GB of host KV cache per node. The matrix budgets only 721 GB, so results report a DRAM allocation the run exceeds. hicache-size: 200 is a per-rank size in GB, and there are 4 TP ranks. Unlike the sibling GB200 hicache recipe, no override sets TOTAL_CPU_DRAM_GB, so validate_recipe never compares the two. Fix: in all 4 dram overrides, keep the host pool within the per-node cap. Use at most 180 GB per rank, or a hicache-ratio. Also pin TOTAL_CPU_DRAM_GB: '721' so the binder rejects a mismatch. Same pattern at 4 sites (agentic.yaml:140, :158, :179, :200).
Why this was flagged
Every master point with kv-offloading: dram in nvidia-master.yaml:5751-5756 sits under dram-utilization: 0.80 on runner cluster:gb200-nv. runners.yaml:579 sets available-cpu-dram-mib: 860_160 (about 902 GB). agentic_dram_offload_gb (infx/matrix/generate.py:475) turns that into total-cpu-dram-gb = 721 for a 4-GPU node. Each of the four new overrides (agentic.yaml:140, 158, 179, 200) sets hicache-size: 200. SGLang allocates that host pool once per TP rank, so 4 ranks hold about 800 GB. That is 79 GB over the budget. Results record allocated_cpu_dram_gb=721 (infx/results/agentic/init.py:205), but the run uses more than that, and it risks host OOM at high concurrency. On the base branch these points ran with KV_OFFLOADING none and no host pool. The safeguard is validate_recipe (infx/srt_slurm/single_node.py:146-148), which compares TOTAL_CPU_DRAM_GB only when the recipe sets it. These overrides omit it, unlike dsv41flash/sglang/gb200-fp4-mtp/agentic.yaml:205, so the check never runs.
Verification: The four new overrides each set hicache-size: 200 with tensor-parallel-size: 4 (agentic.yaml:140, 158, 179, 200). So one node pins about 4 x 200 = 800 GB of host KV. generate.py:486-498 computes a budget of 721 GB. No guard catches the mismatch. single_node.py:145-147 checks TOTAL_CPU_DRAM_GB only if name in workload, and none of the four overrides sets it.
| - { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 64], srt-recipe: agentic.yaml } | ||
| # High-Throughput TP4 + EP4 (MegaMoE) with DRAM KV offload. | ||
| - { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64, 72], srt-recipe: agentic.yaml } |
There was a problem hiding this comment.
🔴 Operators can no longer pick either conc-64 DRAM point with test-config --exp-names. The TP4 and TP4+EP4 arms now produce the same experiment name. Single-node agentic names are built from tp, conc, offload and spec only, with no ep (generate.py:1152-1156). So conc 64 in both new arms yields minimaxm31flash_tp4_conc64_kvdram-hicache_spec-mtp twice. Fix: keep each arm's conc-list disjoint, as nvidia-master.yaml:1520 asks, e.g. by dropping 64 from one arm and its matching override. Otherwise, add ep to the generated name.
Why this was flagged
Both new search-space rows sit under dram-utilization 0.80: tp 4 at conc [32, 64] (5754) and tp 4, ep 4 at conc [64, 72] (5756). _agentic_entries builds exp_name as f"{model_code}_tp{tp}conc{conc}{agentic_kv_offload_suffix(...)}_spec-{spec_decoding}" (generate.py:1152-1156), and ep is not in it. Both conc-64 rows therefore get exp-name minimaxm31flash_tp4_conc64_kvdram-hicache_spec-mtp. filter_exp_names (generate.py:1562-1580) needs exactly one match per name. It raises "Experiment name(s) matched multiple rows", so the sharding EVALS.md:109-112 describes cannot select either point. On the base branch the EP4 arm ran conc 64/128 with kv-offloading none and the TP4 arm stopped at conc 32, so every name was unique. Nothing checks that names are unique; only the comment at nvidia-master.yaml:1520 states the convention.
Verification: Both new arms in nvidia-master.yaml are tp 4, kv-offloading dram with backend hicache, and both list conc 64 (5754, 5756). generate.py:1152-1156 builds the exp_name without ep, so both rows get minimaxm31flash_tp4_conc64_kvdram-hicache_spec-mtp. filter_exp_names (1562-1580) raises on the duplicate. This is a regression: on the base the arms share no conc.
| KV_OFFLOADING: dram | ||
|
|
||
| override_tp4_ep4_c64: | ||
| # TP4 + EP4 (MegaMoE); prefill graphs capped at 16384 on sm_100. |
There was a problem hiding this comment.
🟡 nit (optional): The new comment above the EP4 override group is a section header naming the topology, which AGENTS.md's "No header comments" rule forbids. Fix: drop the comment, or move the sm_100 reason onto the cuda-graph-max-bs-prefill: 16384 line in each override as a line comment.
Why this was flagged
The diff adds a standalone comment at agentic.yaml:165, "# TP4 + EP4 (MegaMoE); prefill graphs capped at 16384 on sm_100.", followed by a blank line and the override_tp4_ep4_dram_c64 and override_tp4_ep4_dram_c72 blocks. AGENTS.md:64 forbids comments above a recipe, override or section that describe the topology or config. AGENTS.md:66 allows only comments that explain a specific line or block. The "TP4 + EP4 (MegaMoE)" half describes topology. The 16384 explanation belongs on the cuda-graph-max-bs-prefill line (agentic.yaml:176 and :197). This is a convention issue, not a runtime one.
Verification: The trigger is just merging the diff as it stands. The diff adds a standalone line, # TP4 + EP4 (MegaMoE); prefill graphs capped at 16384 on sm_100., followed by a blank line, directly above the override_tp4_ep4_dram_c64 / override_tp4_ep4_dram_c72 group (agentic.yaml around line 165). AGENTS.md:66 allows only "A comment explains a specific line or block". Nothing breaks at runtime.
填写 perf-changelog 的 pr-link。
…-sglang-agentx-update # Conflicts: # inferencex-e2e/perf-changelog.yaml
DRAM offload 保持在分配额度内，exp-name 保持唯一。
ispobock
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40176938051 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40176938051 |
…-sglang-agentx-update # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/use 40176938051 |
|
@ispobock staged run 40176938051. This run remains available across future |
Update the GB200 SGLang AgentX configs: add TP4 points with DRAM KV offload, enable it on the TP4 + EP4 points, and adjust the memory settings.
中文
更新 GB200 SGLang AgentX 配置：新增启用 DRAM KV offload 的 TP4 点，TP4 + EP4 的点也启用 DRAM KV offload，同时调整显存相关设置。
AI model disclosure
Prepared with Claude Code using
claude-opus-5-5. Reviewed by the human author before submission.