FEAT: add world model generation support - #5414

Code Review

This pull request introduces support for experimental "World" models (such as Matrix-Game-3.0-5B, HY-WorldPlay-5B, and Astra) in Xinference, adding the necessary RESTful API endpoints, client-side handlers, frontend UI panels, and documentation. The review feedback highlights critical issues that need to be addressed: correcting the frontend JSON validator to return a Promise as required by Ant Design, replacing non-existent/hallucinated Python package versions in the virtualenv specifications with valid PyPI releases, and generating a fallback unique request ID in the API to prevent concurrent task tracking collisions.

This PR adds a first-class experimental world model category for Matrix-Game 3.0, HY-WorldPlay, and Astra, including declarative model/engine registration, generic launch integration, a unified REST and sync/async generation API, frontend workflows, progress plumbing, and documentation. The public path is coherent for an initial CUDA/PyTorch integration, but it introduces unresolved ownership boundaries for subprocess cancellation, the complete runtime dependency/cache closure, and large-media ingress/egress and serving. Blocking: yes — recommended event: COMMENT

Since the previous review at 16cd8bae2c6609b56f27b34640f14849128e8dae, commit a80bedd71730f9698a9966dbe356e670f4ad787a (fix: satisfy world model type checks) changed only two files with three additions and three deletions. It widened Progressor.request_id to Optional[str] and renamed the worker's world cache-manager local to world_cache_manager; it does not change world generation control flow, dependency loading, media transport, or error handling.

Independent approach verdict

acceptable-with-reservations. The separate world category is a reasonable fit for these heavy, CUDA-bound adapters: the registry, generic launch flow, actor boundary, unified VideoList response, and capability-driven UI reuse existing Xinference patterns. The approach is not yet a stable operational contract because request-owned subprocess cleanup, full dependency ownership/offline behavior, and an artifact boundary for long media are left implicit; those boundaries are reflected in G1-G3 below. Round 0 also identified the engine registry's nominal substitution boundary; the confirmed design concern is retained as G14 rather than being inferred from the number of local bugs.

Prior findings checklist

All four prior roots are WAIVED under the qinxuye rebuttal rule and are not open findings:

• WAIVED — frontend validator contract: frontend/src/components/pages/running-model-detail/panels/form-panels.tsx:492, root comment 3836594047, reply 3836600998. The reply establishes that the in-tree FormField contract is (value) => boolean, not Ant Design's Promise validator contract.
• WAIVED — Matrix dependency pins: xinference/model/world/model_spec.json:50, root 3836594050, reply 3836601020. The reply says the versions are published releases matching the pinned upstream requirements.
• WAIVED — HY dependency pins: xinference/model/world/model_spec.json:113, root 3836594056, reply 3836601000. The reply says the versions are published releases matching the pinned HY requirements.
• WAIVED — optional request_id: xinference/api/restful_api.py:3094, root 3836594066, reply 3836601002. The reply correctly notes that CancelMixin ignores None and that optional IDs match existing image/video APIs; this is distinct from the documentation discoverability concern in G13.

The complete raw history exports contain three review records (only review 5000677408 has a body), zero conversation comments, and eight inline records consisting of those four roots and four replies. No linked issue or PR discussion exists. Every G1-G14 group below was independently confirmed against current code and the base commit; no group has a matching prior occurrence, author waiver, or tracking reference.

Confirmed major findings

G1 — major: world subprocesses outlive abort, disconnect, and teardown

Occurrences/evidence: xinference/model/world/model.py:291-306 creates a plain local subprocess.Popen, drains it synchronously in the thread, and has no request registry, process group, cancellation hook, escalation, or guaranteed reap. Matrix, HY, and Astra invoke it at model.py:496-501, model.py:598-603, and model.py:797-803; the abstract/request plumbing is at model.py:350-359. ModelActor.abort_request() at xinference/core/model.py:903-920 finds no world-model abort hook, while _call_wrapper runs the synchronous method through asyncio.to_thread under the actor lock at core/model.py:751-770. The REST cancellation path at xinference/api/restful_api.py:3106-3110 only reports cancellation. Cancelling the awaiting task therefore does not stop the thread or torchrun descendants, and the actor lock/request count can be released while GPU work continues.

Trigger/impact: Abort or disconnect during Matrix, HY, or Astra generation, or terminate a model while a runner is active. The server can report cancellation, accept overlapping work, release logical/GPU capacity, or remove the actor/subpool while torchrun descendants still consume devices, causing overlap, OOM, and orphaned processes.

Fix: Make each request own a process group/session keyed by request_id; terminate the complete group on abort, disconnect, model stop, and actor teardown, wait with a bounded grace period, escalate to kill, and reap descendants before releasing the actor lock, request capacity, device reservation, or subpool. Add child-process cancellation/teardown coverage rather than mocking _run_command only.

History/status: Confirmed canonical root G1 (Round 0 C1 + R1-02); no matching history root or waiver.

G2 — major: source checkout and auxiliary weights bypass the managed artifact lifecycle

Occurrences/evidence: xinference/model/world/core.py:165-167 sends only the primary model through CacheManager. WorldModel.load() unconditionally performs _ensure_source_checkout() at xinference/model/world/model.py:66-147,232-244, and HY/Astra call direct hub snapshot_download() for auxiliary Wan weights at model.py:246-263,518-526,637-646. These operations occur during model_ref.load() after the worker's cancellable primary download/status stage. The specs keep GitHub source_url values (model_spec.json:25-27,84-87,164-167) even for ModelScope variants, and use mutable master revisions for ModelScope model/auxiliary snapshots (model_spec.json:58-60,131-135,200-208).

Trigger/impact: Launch from an apparently cached ModelScope configuration or an air-gapped worker with an empty world_code/auxiliary cache. The launch still needs a GitHub checkout and direct hub downloads, while progress, cancellation, cache status, offline preparation, and remove-cache cover only the primary artifact. A failed load can leave the primary cache marked ready while unmanaged source/auxiliary trees remain or accumulate.

Fix: Represent the complete runtime dependency closure—primary weights, auxiliary weights, and pinned adapter source/package—in a managed manifest with immutable revisions, deterministic cache keys, allow-patterns, ownership, and cleanup policy. Prefetch and validate the closure before actor creation, expose aggregate progress/status, make offline mode local-only for every entry, make cancellation and remove-cache operate on the closure, and replace mutable ModelScope revisions. If Git checkout remains, expose and validate a documented staging workflow; otherwise package the pinned source.

History/status: Confirmed canonical root G2 (Round 0 C2 + R1-07); no matching history root or waiver.

G3 — major: the public media protocol materializes large assets and returns worker-local paths

Occurrences/evidence: xinference/api/schemas/requests.py:111-118 carries media as unrestricted strings inside JSON. Clients and the frontend read complete files into data URLs (xinference/client/restful/restful_client.py:717-755, async_restful_client.py:802-842, and frontend/src/components/pages/running-model-detail/utils.ts:143-155), while the worker accumulates remote chunks or decodes the complete base64 payload before writing a temporary file at xinference/model/world/model.py:161-199. The check is after materialization and there is no output-size limit. _make_response() at model.py:313-329 reads/encodes the full MP4 for b64_json, or moves it to an absolute XINFERENCE_WORLD_DIR path at model.py:315-319; no world GET/media-serving route, worker proxy, artifact ID, ownership, TTL, or cross-worker retrieval boundary exists. The frontend defaults world generation to b64_json at frontend/src/components/pages/running-model-detail/capability-config.tsx:209-218, while HY defaults to 961 frames at model_spec.json:77-82.

Trigger/impact: Long image/video inputs or default long HY output. Whole JSON/base64 payloads and actor serialization create multiple full-size memory copies; url is a producer-worker filesystem path that a browser or another worker cannot retrieve as a public URL, and URL outputs accumulate without cleanup.

Fix: Introduce an explicit artifact boundary: accept multipart/streaming uploads or authenticated artifact IDs, enforce byte limits while streaming, and pass references/streams through REST and actor boundaries. Return a supervisor/object-store or authenticated worker-proxy URL with ownership and bounded TTL/cleanup; keep inline base64 only for explicitly requested small media with an egress cap.

History/status: Confirmed canonical root G3 (Round 0 C3); no matching history root or waiver.

G4 — major: media references permit arbitrary local-file reads and SSRF

Occurrences/evidence: xinference/model/world/model.py:157-159 treats any existing string as a server-local file and returns its real path without an allow-root, symlink, MIME, or size check. The HTTP branch at model.py:161-175 accepts any HTTP(S) URL, uses requests.get(..., stream=True, timeout=30), follows redirects by default, never classifies resolved destinations, and retains chunks in memory; local files bypass the 512 MiB cap. The public schema/route at xinference/api/schemas/requests.py:111-118 and xinference/api/restful_api.py:3075-3116 forwards caller strings unchanged under the ordinary inference access scope.

Trigger/impact: Supply an existing worker path such as /etc/passwd, a loopback/RFC1918/link-local/metadata URL, or an external URL redirecting to one. The worker reads arbitrary local files or probes/fetches internal services; remote responses can consume substantial memory and local sources bypass the intended size limit.

Fix: Remove public server-path semantics. For intentionally supported remote URLs, use a hardened fetcher that rejects unsafe schemes/destinations/ports/credentials, resolves and revalidates every redirect hop (including IPv4/IPv6 private, loopback, link-local, reserved, and metadata ranges), enforces an absolute deadline and cumulative byte cap, and streams directly to a bounded temporary file. Apply the same cap to every source type.

History/status: Confirmed canonical root G4 (R1-01); no matching history root or waiver.

G5 — major: async world generation inherits aiohttp's five-minute default timeout

Occurrences/evidence: The inherited constructor at xinference/client/restful/async_restful_client.py:79-86 sets self.timeout = ClientTimeout(total=1800) but creates ClientSession without passing that timeout. The new AsyncRESTfulWorldModelHandle at async_restful_client.py:802-842 posts at :830-834 without a per-request timeout, so it uses aiohttp's approximately 300-second default. The server route awaits the non-streaming world runner (xinference/api/restful_api.py:3095-3105), which runs a synchronous Popen path through asyncio.to_thread; a client timeout therefore leaves server work running, compounding G1. The async test setup at xinference/client/tests/test_world_handle.py:55-79 uses __new__/mocks and never constructs the real session.

Trigger/impact: A generation lasting over roughly five minutes. The client times out even though the configured handle intends 1,800 seconds and the server-side GPU subprocess continues, producing a misleading failure and possible overlap.

Fix: Pass timeout=self.timeout when constructing the inherited session or explicitly on the world POST, and add a constructor-level test that creates/closes the real handle and asserts the effective session/request timeout. The constructor omission is pre-existing, but the new world handle directly depends on it; this remains a finding for this PR.

History/status: Confirmed canonical root G5 (R1-03), directly impacted pre-existing defect; no matching history root or waiver.

G6 — major: CPU-only engine discovery falsely advertises an installable PyTorch world engine

Occurrences/evidence: xinference/model/world/engine.py:25-35 returns the same false result for missing torch and missing CUDA, while _normalize_match_result in xinference/model/utils.py:76-90 classifies the tuple as dependency_missing. All world specs carry a matching #engine# marker (model_spec.json:30,98,183), so the virtualenv override path at model/utils.py:2333-2337 can advertise virtualenv_required on a CPU-only worker. The launch path then performs virtualenv/cache setup before the load-time CUDA guard at xinference/model/world/model.py:232-243 fails.

Trigger/impact: Query or launch a world model on a CPU-only deployment with virtualenv support enabled (the default). The frontend/cluster advertises PyTorch, prepares an environment and primary weights, and fails late rather than rejecting the host before expensive setup.

Fix: Separate installable-library checks from non-installable host capability. Return a structured host-incompatible result or enforce CUDA in the engine match/discovery path so virtualenv markers can bypass only missing dependencies, never CUDA/platform requirements. Apply the same gate in launch before cache/venv preparation and add CPU-only discovery/early-rejection coverage.

History/status: Confirmed canonical root G6 (R1-04); no matching history root or waiver.

G7 — major: HY overloads --input, allowing prompt-as-file and prompt@image reinterpretation

Occurrences/evidence: xinference/model/world/model.py:558-576, especially :561-562, passes arbitrary public prompt text as --input and executes from the HY checkout at :598-603. At the pinned HY source revision, wan/generate.py treats --input as either a text-file path or prompt@image_path; a prompt equal to README.md expands into every nonblank line in the checkout file, while a prompt containing @ is split and may override the explicit image path. The REST schema/handler accepts arbitrary prompt strings (requests.py:111-118, restful_api.py:3097-3104), and existing tests cover only adapter argument forwarding.

Trigger/impact: Prompt README.md, another checkout filename, or text such as an email/handle. One API request can become hundreds of expensive 5B generations, or the prompt can be truncated and an unintended image selected.

Fix: Do not expose the upstream overloaded parser to caller text. Add a pinned adapter-owned wrapper or direct structured call with separate prompt and image arguments, invoke exactly one prediction, and add regressions for an existing filename, an @ prompt, and explicit image preservation.

History/status: Confirmed canonical root G7 (R1-05); no matching history root or waiver.

G8 — major: invalid response_format is rejected only after full GPU generation

Occurrences/evidence: Matrix, HY, and Astra pop response_format at xinference/model/world/model.py:438, :554, and :709 without validating it. Each then materializes inputs and runs the expensive subprocess (model.py:439-501, :556-603, :710-803); the only format check is _make_response() at model.py:313-329, after a complete MP4 exists. The permissive nested request bags at xinference/api/schemas/requests.py:111-119 and REST forwarding at restful_api.py:3075-3116 leave invalid values reachable.

Trigger/impact: Send an unsupported value such as response_format: "base64" in generation_config or extra_body. The request spends the full GPU generation and temporary output lifecycle before raising, wasting minutes and resources.

Fix: Add one shared validator accepting exactly url and b64_json, invoke it immediately after merged defaults/config and before TemporaryDirectory, input materialization, or runner launch, while retaining _make_response as a defensive guard. Add Matrix/HY/Astra tests proving the runner is not called for invalid values.

History/status: Confirmed canonical root G8 (R1-06); no matching history root or waiver.

G9 — major: Matrix automatic multi-GPU FSDP flags are disabled by the pinned runner

Occurrences/evidence: xinference/model/world/model.py:483-487 auto-enables dit_fsdp and t5_fsdp when _gpu_count() > 1, but no default ulysses_size is supplied. The pinned Matrix runner defaults ulysses_size to 1 and explicitly disables both FSDP flags when it is not greater than one. The adapter still launches one torchrun rank per assigned device (model.py:265-278), so each rank can load/compute a full model instead of sharding; the pinned upstream test invocation supplies ulysses_size explicitly.

Trigger/impact: Assign more than one GPU to Matrix without an explicit ulysses_size. Intended multi-GPU sharding is silently absent, causing duplicated compute or OOM on configurations that rely on sharding.

Fix: When auto-enabling FSDP, set ulysses_size to the assigned visible GPU count if the caller did not provide it; reject conflicting values such as 1, or remove the automatic flags and clearly reject/document unsupported multi-GPU operation. Add a two-GPU command/runner mapping regression.

History/status: Confirmed canonical root G9 (R1-08); no matching history root or waiver.

Confirmed minor findings

G10 — minor: model-side validation errors become HTTP 500

Occurrences/evidence: xinference/api/restful_api.py:3106-3116 maps only cancellation specially; the blanket except Exception at :3111-3116 turns adapter ValueErrors from xinference/model/world/model.py:150-205,314-348,401-435,540-552,669-707 into HTTP 500. The direct API tests do not exercise the actual route exception/status contract.

Trigger/impact: Missing required image, unsupported video, invalid frame/camera/options, duplicate bags, or invalid response format is reported as an internal server error instead of a stable caller-facing 400/422.

Fix: Use a distinct validated user/configuration error path and map it to 400/422 before the generic branch; preserve 409 for cancellation, 429 for rate limits, and 500 for runner/download/internal failures. Add route-level status tests for representative user and internal failures.

History/status: Confirmed canonical root G10 (R1-09); no matching history root or waiver.

G11 — minor: Astra's selector silently overwrites advanced extra_body.cam_type

Occurrences/evidence: frontend/src/components/pages/running-model-detail/capability-config.tsx:184-205 parses advanced JSON and then unconditionally assigns extraBody.cam_type = cameraMotion at :189-193; the selector default is 1 at :209-214. Thus an advanced { "cam_type": 4 } is replaced by 1 unless the selector is changed, while putting cam_type in generation_config creates a duplicate-key error after the UI still injects extra_body.cam_type.

Trigger/impact: A caller enters a non-default camera trajectory in the advanced model kwargs but leaves the friendly selector at its default. The request silently runs the wrong trajectory; an alternative bag produces an opaque backend error.

Fix: Define one source of truth: preserve explicit extra_body.cam_type, inject the selector only when neither bag specifies it, and surface a user-visible conflict if both bags contain it. Add focused transform coverage for advanced, selector, and conflict cases.

History/status: Confirmed canonical root G11 (R1-10); no matching history root or waiver.

G12 — minor: HY zero-exit/no-video failures discard the runner traceback

Occurrences/evidence: _run_command() only tails runner.log for a nonzero exit at xinference/model/world/model.py:306-311. HY then checks the output count at model.py:604-608; the pinned runner catches per-input exceptions, writes traceback information, and exits zero, so this branch raises only the generic no-video message and deletes the temporary log.

Trigger/impact: A handled HY input/model/download failure returns an opaque HTTP 500 with no actionable child traceback, making diagnosis and correction difficult.

Fix: On the no-video/output-count failure, include a bounded sanitized tail of runner.log/err.txt, or use a wrapper that turns per-input failures into a nonzero/summary status. Add a zero-exit/no-output regression with a logged traceback.

History/status: Confirmed canonical root G12 (R1-11); no matching history root or waiver.

G13 — minor: public documentation omits the progress/cancel and supported offline-source workflow

Occurrences/evidence: doc/source/models/model_abilities/world.rst:10-27 lists fields and a synchronous endpoint but does not explain the caller-generated ID hidden in extra_body/generation_config, progress polling at /v1/requests/{id}/progress, or the abort path. It also omits the supported source checkout/auxiliary staging workflow, output URL reachability/retention, and the fact that ModelScope weights still require the GitHub source checkout. The frontend implementation at capability-task-panel.tsx:108-181 depends on this hidden flow, while the source staging is implemented only through model.py:66-147,217-244 and not exposed in world docs/UI.

Trigger/impact: REST/Python users cannot discover progress or cancellation for multi-minute requests, and users can pre-cache weights for an air-gapped deployment yet fail on the undocumented source/auxiliary requirements. This is a documentation root only; it does not duplicate the code findings in G1-G3.

Fix: Add an operational subsection with cURL/Python examples for the request ID placement, progress response/retention, and abort call; document current source/auxiliary pre-staging and ModelScope/GitHub behavior; describe current worker-local url and memory-heavy b64_json semantics until G2/G3 establish a new artifact contract. Keep the optional-ID collision waiver separate.

History/status: Confirmed canonical root G13 (Round 0 documentation concern + R1-12); no matching history root or waiver.

G14 — minor design concern (body-only): the engine abstraction is nominal rather than behavioral

Occurrences/evidence: xinference/model/world/engine.py:25-60 exposes PyTorch* classes that mainly match families and check libraries, while the supposedly engine-neutral WorldModel at xinference/model/world/model.py:232-278 imports torch, requires CUDA, counts devices, and builds torchrun commands. The concrete classes inherit WorldModel before the engine mixin, and Matrix/HY/Astra directly use those PyTorch assumptions. The registry can substitute a complete concrete class at the actor boundary, but it does not provide a reusable runtime contract comparable to the established image/audio/LLM engine layers.

Trigger/impact: Adding another runtime or relying on the comment that runtimes can be added without changing the model contract requires overriding/reduplicating load/device/process behavior and untangling family-specific logic. This is a maintainability/substitutability concern, not a current launch failure.

Fix: Either intentionally keep the first implementation single-engine and remove/rename the generalized indirection until a second runtime exists, or define a real runtime interface and move CUDA/device/process behavior into a PyTorch runtime layer while family adapters own semantic validation and argument mapping. Add a second-engine contract test before advertising substitution.

History/status: Confirmed canonical root G14 (Round 0 design concern); no matching history root or waiver.

Testing and coverage limitations

This follow-up is a static review; no local tests, builds, linters, dependency installation, or runtime reproductions were run. The added tests cover useful registry/config mapping, direct REST forwarding, client body encoding, and Astra progress parsing, but mock or bypass the boundaries where the confirmed risks live. Missing coverage includes process-group cancellation and teardown, complete source/auxiliary cache/offline/remove lifecycle, SSRF/local-file/redirect and streaming size enforcement, public artifact retrieval and retention, construction of the real async session timeout, CPU-only discovery before cache/virtualenv setup, pinned HY input parsing, early response-format validation, two-GPU Matrix flags, route-level 4xx mapping, and frontend Astra transform precedence. The raw Round 1 gap check also found no additional code-quality root, but browser rendering, live multi-worker scheduling, air-gapped launch, and real actor abort behavior remain unexercised.

Blocking status & recommended decision

Blocking is yes because nine confirmed major roots remain. The required event is COMMENT (this PR is authored by qinxuye; no approval or change-request event is recommended).

• xinference/model/world/model.py:291 — major — world Popen/torchrun descendants survive abort/disconnect/teardown and capacity can be released early. [new] Recommended event: COMMENT.
• xinference/model/world/core.py:165 — major — source checkout and auxiliary weights bypass managed cache, offline, cancellation, status, and removal lifecycle. [new] Recommended event: COMMENT.
• xinference/model/world/model.py:317 — major — large media is copied through whole JSON/base64 and url is an unreachable worker-local path. [new] Recommended event: COMMENT.
• xinference/model/world/model.py:157 — major — public media references allow arbitrary local-file reads and SSRF through redirects. [new] Recommended event: COMMENT.
• xinference/client/restful/async_restful_client.py:830 — major — the new async world POST inherits aiohttp's 300-second default instead of the configured 1,800-second timeout. [new] Recommended event: COMMENT.
• xinference/model/world/engine.py:34 — major — CPU-only workers advertise a virtualenv-installable PyTorch engine and fail only after setup/cache work. [new] Recommended event: COMMENT.
• xinference/model/world/model.py:561 — major — HY interprets arbitrary prompts as filenames or prompt@image inputs. [new] Recommended event: COMMENT.
• xinference/model/world/model.py:438 — major — invalid response formats consume a full GPU generation before rejection. [new] Recommended event: COMMENT.
• xinference/model/world/model.py:483 — major — automatic Matrix FSDP flags are disabled because ulysses_size remains 1. [new] Recommended event: COMMENT.

The five confirmed minor roots (G10-G14) are reportable but do not independently change the blocking status.

This PR adds world as a first-class Xinference model type for Matrix-Game 3.0-5B, HY-WorldPlay-5B, and Astra, with catalog/engine/worker launch integration and one REST generation protocol. It also adds sync/async client handles, progress/abort plumbing, ability-driven frontend forms/results, API-key display, and operator documentation for the initial text/image flows.
Blocking: yes — recommended event: COMMENT

Update summary

Since the previous reviewed commit 16cd8bae2c6609b56f27b34640f14849128e8dae, two commits landed: a80bedd7 (fix: satisfy world model type checks) and 43eee4e2 (fix: harden world generation runtime). The update hardens runner process-group registration/TERM-KILL/reaping, adds shared response-format validation and model-side HTTP status mapping, and tightens normal multi-GPU defaults and host checks. Those changes close G4, G5, G7, G8, G10, and G12; residual portions of G1, G6, G9, and G11 remain, and the new findings below are still actionable.

Approach verdict

Acceptable with reservations. The declarative family/engine manifest and reuse of the existing worker/GPU/virtualenv machinery, REST/client surface, progress tracking, and ability-driven UI are coherent for the stated first-version scope. The reservations below are concrete lifecycle, validation, allocation, frontend, documentation, and boundary-test gaps; no standalone architecture or module-boundary finding is being raised under the qinxuye review policy.

Prior-findings checklist

Carried prior roots

• G1 — PARTIAL, major (still open) — Prior source 3838489762, reply 3838516872. Registered Popen process groups now receive TERM/KILL and are reaped, but cancellation before _running_processes registration or while waiting for _runner_lock still returns NO_OP; the to_thread worker can later start the GPU runner and release request capacity before cleanup.
• G4 — FIXED — Prior source 3838489773, reply 3838527709. REST now accepts only bounded typed base64 data URLs and the materializer has no public local-path or HTTP/SSRF branch.
• G5 — FIXED — Prior source 3838489776, reply 3838527783. The async world POST passes the configured timeout.
• G6 — PARTIAL, minor (still open) — Prior source 3838489779, reply 3838516864. Direct host discovery/factory checks now reject CPU-only hosts, but launch-side environment/subpool work can precede the final host check.
• G7 — FIXED — Prior source 3838489782, reply 3838516886. HY uses the structured adapter/runner call rather than an overloaded public input.
• G8 — FIXED — Prior source 3838489783, reply 3838516861. Shared response-format validation runs before media materialization and runner startup for the adapters.
• G9 — PARTIAL, minor (severity adjusted) — Prior source 3838489785, reply 3838516862. Normal multi-GPU defaults now derive ulysses_size and FSDP settings from the assigned count, but explicit invalid or mismatched values still reach the runner.
• G10 — FIXED — Prior source 3838489787, reply 3838516868. Model-side ValueError and cancellation paths now map to the intended HTTP statuses; route-boundary parse failures are the separate N1 finding below.
• G11 + N7 — PARTIAL, minor (one canonical root) — G11 prior source 3838489790, reply 3838527825; N7 is the current Round 1 occurrence. Conflicting Astra cam_type values are now rejected, but the frontend transform rejection is not caught or shown to the user.
• G12 — FIXED — Prior source 3838489793, reply 3838516865. Direct runner exceptions now produce a nonzero exit and bounded log-tail diagnostics.
• G13 — PARTIAL, minor — Prior source 3838489795, reply 3838527881. The new page documents the main request/progress/abort/media/source flow but omits several response, retention, worker-local, and staging details.

Waived by explicit qinxuye rebuttals (not open and not re-reported)

• Gemini validator contract — WAIVED — 3836594047 → reply 3836600998.
• Gemini Matrix dependency pins — WAIVED — 3836594050 → reply 3836601020.
• Gemini HY dependency pins — WAIVED — 3836594056 → reply 3836601000.
• Gemini optional request_id — WAIVED — 3836594066 → reply 3836601002.
• G2 managed dependency/cache/source lifecycle — WAIVED — 3838489765 → reply 3838516856, with conversation context 5386045967.
• G3 media artifact/transport/lifecycle — WAIVED — 3838489771 → reply 3838516860, with conversation context 5386045967.
• G14 engine abstraction/module boundary — WAIVED — body source 5002371552, with conversation context 5386045967.

Dropped

• N8 custom world registration — DROPPED — The public supervisor map omits world and rejects unsupported registration before the worker, so the alleged public 500 path is not reachable.
• N10 hidden config namespaces — DROPPED — The documented generation_config/extra_body split and deterministic request_id precedence do not establish a concrete compatibility failure; this remains preference/speculation.

No linked issue or PR discussion existed beyond the exported PR history above.

Confirmed findings — major

1. G1 [prior] — Cancellation can miss pre-registration work — major, xinference/model/world/model.py:294.
   The process is inserted into _running_processes only after the runner lock and Popen. Because the actor invokes synchronous world_generate() through asyncio.to_thread, cancelling the actor await does not cancel preprocessing or lock wait; abort_request() can observe no process and return NO_OP, after which the expensive GPU process starts anyway. Register a pending per-request cancellation marker before preprocessing/lock acquisition, check it immediately before and after spawn, and hold request capacity until the worker thread and process group are reaped.

2. N2 [new] — Synchronous local-media encoding blocks the async event loop — major, xinference/client/restful/async_restful_client.py:818.
   encode_world_reference() performs local-file read() and base64 encoding in the coroutine before its first await. With the accepted large-media limit, one request can block every coroutine sharing the loop and create another full-size buffer. Preserve already encoded values, but use await asyncio.to_thread(encode_world_reference, ...) for local files or a bounded streaming upload, and add a loop-responsiveness test.

3. N3 [new] — Reset/unmount abandons the GPU request — major, frontend/src/components/pages/running-model-detail/panels/capability-task-panel.tsx:149.
   The submit path creates a request ID but does not retain it or an AbortController; reset() and unmount at frontend/src/components/pages/running-model-detail/panels/capability-task-panel.tsx:183-195 only invalidate polling. A discarded Matrix/HY/Astra generation can therefore continue consuming GPU time after the user leaves the panel. Track active request/cancellation state and best-effort call the model abort endpoint on reset, ability changes, and unmount.

4. N5 [new] — use_async_vae targets an unreserved GPU — major, xinference/model/world/model.py:538.
   The option is exposed and forwarded as a runner flag, while _torchrun_command() launches one process on every assigned GPU and the pinned runner starts async VAE on device world_size. One assigned GPU therefore targets cuda:1; with N assigned GPUs it uses unreserved device N. Reserve an extra GPU and launch torchrun on N-1, or reject the option until the runner contract is fixed, with single- and multi-GPU command-contract tests.

5. N6 [new] — Model-specific config validation is too weak — major, xinference/model/world/model.py:606.
   The allowlist checks option names but not strict per-adapter types, ranges, or relationships before media materialization and runner startup. Zero/negative/float frame or chunk values, malformed poses, nulls and boolean strings, invalid Matrix ulysses_size, and inconsistent Astra frame/MoE values can be coerced or fail late as runner errors/500s. Add strict schemas with real boolean parsing and relationship checks before spawn; keep this validation contract separate from the G9 resource-mapping fix.

Confirmed findings — minor

1. G6 [prior] — CPU-only launch still performs side effects before the final capability check — minor, xinference/core/worker.py:3584 (the changed world launch branch; affected ordering is xinference/core/worker.py:3692-3763,3818-3849).
   create_model_instance() performs the world host check only after virtualenv preparation and, for multi-worker launches, subpool creation. A CPU-only launch can therefore leave an environment or transient subpool before failing. Move the capability check ahead of those side effects and add cleanup coverage for the empty-environment/subpool case.

2. G9 [prior] — Explicit Matrix FSDP/ulysses values remain late-failing — minor, xinference/model/world/model.py:520.
   The normal multi-GPU default is fixed here, but explicit non-integral, mismatched, or invalid ulysses_size/FSDP values are only stringified into runner arguments and can fail in torchrun or pinned assertions after launch. Validate integer/divisibility and FSDP/ulysses relationships before constructing the command.

3. N1 [new] — Malformed world request bodies become HTTP 500 — minor, xinference/api/restful_api.py:3076.
   request.json() and WorldGenerationRequest parsing happen before the try beginning at line 3111. Malformed JSON, non-object/missing fields, and null or wrong-typed configuration dictionaries bypass the 4xx ValueError branch and reach generic 500 handling. Let FastAPI bind the typed body or catch JSON/Pydantic validation errors here and return consistent 4xx responses; add an ASGI route test.

4. G11/N7 [current merged root] — Frontend transform rejection is silent — minor, frontend/src/components/pages/running-model-detail/capability-config.tsx:196.
   This transform now deliberately raises for duplicate/conflicting Astra cam_type values, but the caller at frontend/src/components/pages/running-model-detail/panels/capability-task-panel.tsx:149-177 has a success handler and finally only, with no rejection handler. FileReader failures and this conflict can clear loading without a user-visible error or can become unhandled rejections. Catch transform failures and route them through the panel's existing error state.

5. G13 [prior] — Operational documentation omits important response and lifecycle contracts — minor, doc/source/models/model_abilities/world.rst:104.
   Please add the exact VideoList.data[0] response shape, progress payload and five-minute retention behavior, worker-local URL/no-cleanup semantics, raw b64_json form, and the fact that ModelScope still requires the pinned GitHub adapter checkout. The current caveat is not enough for operators to size storage or stage sources safely.

6. N4 [new] — World permissions cannot be edited in the API-key dialog — minor, frontend/src/components/pages/api-key-management/index.tsx:55.
   Adding world to the display set makes existing rows render, but the edit dialog still derives checkboxes from MODEL_TYPE_OPTIONS at frontend/src/components/pages/api-key-management/utils.ts:50-57, which has no world entry. Administrators therefore cannot create or edit a world-only key through the UI. Add one shared world option to the dialog source and cover create/edit round-tripping.

7. N9 [current test-boundary root] — Added tests do not exercise the public boundaries — minor, xinference/api/tests/test_world_api.py:93.
   This test calls RESTfulAPI.create_world unbound with _Request/_API fakes, so route registration/auth, FastAPI parsing and 4xx behavior, response validation, and transport are not exercised. Client, actor, and frontend tests likewise mock the boundaries implicated by the findings above. Keep the focused unit tests, but add an ASGI route test plus discriminating async-encode, pre-registration-abort, GPU-allocation, and frontend cleanup/error cases.

Testing and coverage limitations

No tests, linters, formatters, builds, or targeted reproductions were run locally. The review relied on current-source and base-commit inspection; CI remains the execution authority. The test-boundary limitations above are the remaining concrete N9 gap, not a duplicate of the G1 or G11/N7 implementation findings.

The Simplification Lens was unavailable because the review-spark run failed with usage_limit_reached; no fallback scan was run, so no Simplification opportunities section is included.

Blocking status & recommended decision

Blocking is yes because five confirmed major roots remain. Minor roots never block. For a qinxuye-authored PR, the required event is COMMENT, not REQUEST_CHANGES.

Blocking issues:

• xinference/model/world/model.py:294 — major — cancellation before process registration can return NO_OP and later start a cancelled GPU runner. [prior]
• xinference/client/restful/async_restful_client.py:818 — major — synchronous local-media read/base64 encoding blocks the shared async event loop. [new]
• frontend/src/components/pages/running-model-detail/panels/capability-task-panel.tsx:149 — major — reset/unmount leaves the world generation running. [new]
• xinference/model/world/model.py:538 — major — use_async_vae addresses an unreserved GPU. [new]
• xinference/model/world/model.py:606 — major — weak model-specific validation permits coercion and late runner/500 failures. [new]

Recommended event: COMMENT.

1. PR Summary

This PR adds a new world model ability (Text2world / Image2world / Video2world) to Xinference, following the existing category conventions: a spec-class model definition, registry hooks, REST/client surface, and capability-driven frontend UI. It wires three concrete adapters (Matrix-Game, HY-WorldPlay, Astra) that shell out to model-specific inference scripts via subprocess, plus progress reporting, abort support, and documentation. This is a re-review: a prior round left 12 canonical findings, and this round both re-verifies those and does fresh static analysis of the code as it stands at commit 3df40410.

Blocking: yes — recommended event: COMMENT

2. Update Summary

Since the last reviewed commit (43eee4e2), commits a80bedd7, fc68a833, and 3df40410 fixed 11 of the 12 prior findings: pre-registration cancellation handling (G1), event-loop-blocking media encoding (N2), abandoned-GPU-request-on-reset (N3), unreserved-GPU rejection for use_async_vae (N5), stronger per-model config validation (N6), CPU-only host-check ordering (G6), late-failing Matrix FSDP/ulysses validation (G9), malformed-body 500→400 (N1), silent frontend transform rejection (G11/N7), doc gaps (G13), and missing world permissions in the API-key dialog (N4). One finding (N9, test coverage of public boundaries) is only partially addressed — see below.

3. Approach Verdict

Acceptable-with-reservations. The world category conforms well to Xinference's existing conventions — spec-class shape, registry pattern, REST/client surface, capability-driven UI, and teardown via existing ModelActor hooks all follow established patterns. The concrete concerns below are correctness, security, and coverage gaps rather than design objections; larger structural questions (engine_family duplication, the git-clone-execution model, the two-config-bag shape) were already raised and explicitly waived in the prior round.

4. Prior-Findings Checklist

5. Confirmed Findings — Major

[new] 1. xinference/model/world/__init__.py:34-39 + xinference/core/worker.py:2221-2237 — masking TypeError on custom-model registration failure
register_world/unregister_world raise NotImplementedError for custom models, but worker.py's register_model catches this via except Exception and calls unregister_fn(model_spec.model_name, raise_error=False). unregister_world's signature is (model_name, version=None) — no raise_error param — so this raises TypeError: unregister_world() got an unexpected keyword argument 'raise_error', masking the original error and surfacing as HTTP 500 instead of a clean 4xx. register_video/unregister_video are no-op pass stubs, so video is unaffected — world is uniquely broken here. Fix: accept **kwargs (or raise_error) in unregister_world, or raise ValueError instead of NotImplementedError in both functions. (Note: worker.py:2221-2237 itself is pre-existing code untouched by this diff — the fix belongs in the world-specific stubs.)

[new] 2. xinference/api/schemas/requests.py:111-118 — WorldGenerationRequest silently drops unknown fields
No Config.extra = "forbid" override (Pydantic v1-compat default is Extra.ignore). Sibling TextToImageRequest (same file, lines 56-63) has a top-level response_format field; a caller who sends {"model":..., "prompt":..., "response_format":"b64_json"} to the world endpoint by analogy gets response_format silently dropped, falls back to "url", and gets an unusable worker-local filesystem path with no warning. Also user: Optional[str] is accepted but never read anywhere in the handler — dead field. Fix: set extra="forbid" on WorldGenerationRequest; remove or wire up user.

[new] 3. xinference/api/restful_api.py:3072-3079 — unbounded body read before size validation (DoS)
create_world calls await request.json() on the full body with no size limit before any validation runs. _MAX_INPUT_BYTES (512 MiB, model.py:42) is only checked deep inside _materialize_reference, after the full JSON is parsed/buffered in the API process and shipped across the actor RPC boundary to the worker. A ~683 MB base64 payload can stall/OOM the API process before any check runs. This pattern is new relative to siblings (create_images/videos take pure text; image edits/variations stream via UploadFile multipart). Fix: enforce a request body size cap before request.json().

[new] 4. xinference/model/world/model.py:212,294 (_runner_lock) — process-lifetime lock can starve the shared thread pool
_run_command holds a threading.Lock for the entire subprocess lifetime (5+ min per the PR's own manual test), invoked via asyncio.to_thread from ModelActor._call_wrapper. request_limits defaults to unbounded (float("inf")) for world models, so N concurrent requests to the same model instance each occupy a worker thread of Python's shared default ThreadPoolExecutor, all blocked on the same lock — potentially starving every other asyncio.to_thread call in the process, including ModelActor.stop() teardown and abort_request's termination call. Fix: default request_limits to 1 for world models, or use a dedicated thread pool / async subprocess API instead of asyncio.to_thread + blocking lock.

[new] 5. xinference/core/model.py:751-771 — disconnect with no request_id leaves an unabortable GPU subprocess
request_id is optional on create_world. If the client disconnects, asyncio.to_thread's uncancellable nature plus the shield-based _wait_for_sync_on_cancel logic means the ~5-minute GPU subprocess runs to completion with no abort path — restful_api.py's except asyncio.CancelledError → abort_request(request_id) is gated on if request_id:, and the internal anonymous fallback key (f"anonymous-{uuid4().hex}", model.py:283) is never exposed outside the worker thread. self._lock (one per ModelActor) also stays held the whole time, blocking any other request to that replica. Fix: require or server-side-generate a request_id for every request so it's always abortable.

[new] 6. doc/source/models/model_abilities/world.rst — model-specific config keys almost entirely undocumented
~40+ config keys across the three adapters' _SUPPORTED_CONFIG (Matrix-Game: 22, HY-WorldPlay: 6, Astra: 16 — model.py:485-508/693-700/818-835) go essentially undocumented; only cam_type and one generic num_frames example appear. The API hard-rejects unknown keys. HY-WorldPlay's pose grammar (regex (w|s|a|d|up|down|left|right)-<number>(,...), model.py:743) and Matrix-Game's num_frames == 57 + 40k constraint (model.py:545-552) are especially non-obvious and undocumented. Fix: document the supported key set and constraints per model.

[new] 7. frontend/.../capability-config.tsx:714,744 + xinference/model/world/model.py — progress bar freezes for 2 of 3 models
showProgress: true is set for Text2world/Image2world (Matrix-Game-3.0-5B, HY-WorldPlay-5B), but both models' world_generate call progressor.set_progress(0.02, ...) exactly once with no progress_callback passed to _run_command (model.py:677-685, 800-808) — only Astra has real regex-based progress parsing (model.py:1002-1027,1036). The progress bar freezes at 2% for the whole generation (up to ~296s per the PR's manual test) then jumps to 100% via Progressor.__exit__. Fix: wire real progress parsing for Matrix-Game/HY-WorldPlay, or set showProgress: false for those models until it exists.

[new] 8. frontend/.../panels/capability-task-panel.tsx:277 — Reset-without-abort regression for 9 unrelated capabilities
Removing disabled={loading} from the Reset button (needed for world-model abort-via-reset) was applied globally, not scoped to world. abortActiveRequest() only fires when activeRequestRef.current is populated, which only happens when config.showProgress is true — true for 9 of 18 capabilities. For the other 9 (Generate/LLM, Embed, SpeakerEmbedding, Rerank, Ocr, Docanalyze, Audio2text, Text2audio, Text2music), Reset is now clickable mid-request; the UI silently discards the result client-side but sends no abort — the backend request/GPU/inference slot keeps running unaborted. This is a real regression for pre-existing, unrelated categories. Fix: scope the disabled={loading} removal to config.showProgress capabilities only, or generate a requestId for all capabilities.

[new] 9. Test coverage gaps on new/shared code paths
Confirmed genuinely untested via grep across all three test files:

• _materialize_reference (model.py:153-194) — no coverage of base64 decode failure, malformed data: header, _MAX_INPUT_BYTES pre/post-decode checks, or temp-file cleanup.
• _ensure_source_checkout/_validate_checkout (model.py:56-150) — no coverage of git-init/fetch/checkout, FileLock, stale-revision revalidation, or cleanup-on-exception; tests set model._code_path directly, bypassing this path entirely.
• _make_response's b64_json success path (model.py:394-408) — only the rejection of an invalid response_format is tested; all success-path tests assert on "url" only.
• Concurrent generation via _runner_lock, and WorldModel.stop() — no test issues two overlapping _run_command calls, and .stop() is never called by any test.
• _wait_for_sync_on_cancel in ModelActor._call_wrapper (core/model.py:751-771) — shared code used by every model category, zero coverage at the ModelActor level.

6. Confirmed Findings — Minor

[new] 10. xinference/model/world/model_spec.json + model.py — Video2world is fully-wired but unreachable dead code
No model_spec declares a video2world ability, and all three adapters unconditionally reject video input, yet the PR ships a complete Video2world capability config, a VideoToWorldPanel with a required video upload, i18n strings in 4 locales, and video parameters threaded through client handles and the REST schema. Verified genuinely unreachable today (tab visibility gated correctly by model.model_ability), so it's dead-but-harmless — but it's untested surface that would guarantee a 400 if a future spec ever added video2world without a matching adapter update. Fix: remove until a model supports it, or mark explicitly reserved.

[new] 11. xinference/model/world/model.py:560-566,576,591-592,732-737,884-888 — validation helpers assume defaults always present
Several blocks call _require_int/_require_number/_require_string unconditionally (bare config[key], no .get()) on keys not always guarded by if key in config (e.g. lightvae_pruning_rate at line 576 is guarded in one loop but accessed unconditionally on the next line). Currently harmless because model_spec.json's default_generate_config always supplies these keys and _merge_configs is additive-only. Fragile: a future edit dropping one of these defaults produces a bare KeyError → generic 500 instead of a clean 400. Fix: make _require_* tolerant of missing keys, or add defensive presence checks.

7. Informational (Non-Blocking)

12. Abort endpoint has no per-request ownership check. POST /v1/models/{model_uid}/requests/{request_id}/abort has no ownership/ACL check tying a caller to the request_id they created — any caller with models:read scope who knows/guesses another tenant's request_id for the same model can abort their in-flight generation. This is a pre-existing, repo-wide gap already present for image/audio/video before this PR — this PR only extends the same mechanism to a new category, and does not worsen it. Not blocking; worth a repo-wide follow-up. Separately, doc/source/models/model_abilities/world.rst's use of predictable example IDs like "world-request-1" is a documentation nit (recommend a UUID-style example) rather than a security finding, given the underlying mechanism predates this PR.

8. Simplification Opportunities

• xinference/model/world/engine_family.py:107,142 — the host_result → reason tuple-unwrap block is duplicated verbatim in check_engine_by_model_name_and_engine and check_engine_by_model_name_and_engine_with_virtual_env. Extract a _host_check_reason(host_result, engine_name) helper. Low-risk, confirmed duplication.
• xinference/model/world/model.py:642-671,781-790,962-989 — the value_options dict-to-CLI-argv loop repeats near-identically across MatrixGameModel, HYWorldPlayModel, AstraModel. Extract a shared WorldModel._append_value_options(command, config, mapping). A shared _append_flag_options helper only cleanly covers MatrixGameModel and 2 of Astra's 3 flag-like options — HYWorldPlayModel has no flag_options loop, and Astra's use_camera_cfg emits --use_camera_cfg true (not a bare flag) and must be excluded from any generic flag helper. Recommend extracting _append_value_options only, or _append_flag_options with an explicit exclusion list.
• Investigated and dropped: inlining _wait_for_sync_on_cancel out of _call_wrapper — that wrapper is genuine shared cross-cutting logic (locking, generator conversion, running-task bookkeeping); inlining would duplicate all of that for one caller's 10-line shield branch, which is not simpler.

net: -15 to -20 lines possible

9. Testing and Coverage Limitations

No tests, builds, or lints were run locally for this review; findings are based on static code reading against commit 3df40410, and the coverage gaps noted in finding 9 were confirmed via grep across the existing test files, not by executing them. CI is the execution authority for pass/fail status.

10. Blocking Status & Recommended Decision

Blocking: yes
Recommended event: COMMENT

Blocking issues:

• xinference/model/world/__init__.py:34-39 — major — custom-model unregistration masks the real error behind a TypeError, surfacing as HTTP 500. [new]
• xinference/api/schemas/requests.py:111-118 — major — WorldGenerationRequest silently drops unknown fields (e.g. response_format), unlike sibling TextToImageRequest. [new]
• xinference/api/restful_api.py:3072-3079 — major — unbounded request.json() read before size validation, DoS risk. [new]
• xinference/model/world/model.py:212,294 — major — process-lifetime _runner_lock held via asyncio.to_thread can starve the shared thread pool under concurrent requests. [new]
• xinference/core/model.py:751-771 — major — disconnect on a request without request_id leaves a ~5-minute GPU subprocess unabortable. [new]
• doc/source/models/model_abilities/world.rst — major (docs) — ~40+ model-specific config keys are essentially undocumented despite hard rejection of unknown keys. [new]
• frontend/.../capability-config.tsx:714,744 — major (UX) — progress bar freezes at 2% for Matrix-Game/HY-WorldPlay for the full generation duration. [new]
• frontend/.../panels/capability-task-panel.tsx:277 — major (regression) — Reset button now clickable mid-request for 9 unrelated, pre-existing capabilities without sending an abort. [new]
• Test coverage gaps in _materialize_reference, _ensure_source_checkout, b64_json response path, concurrent _runner_lock use, WorldModel.stop(), and _wait_for_sync_on_cancel — major (coverage). [new]