feat(lm): support Qwen3.8-Flash-Next with bounded SSD inference - #154
Brooooooklyn wants to merge 1 commit into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ab150b88-d43f-4669-bcbe-3cfd47106e5b) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51bb619914
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let cache_limit = CACHE_BYTES; | ||
| let mut s = Self { | ||
| tensors: HashMap::new(), | ||
| gguf: path.extension().is_some_and(|e| e == "gguf"), |
There was a problem hiding this comment.
For a mixed-case filename such as model.GGUF, the public detector treats the path as GGUF because packages/lm/src/model-detection.ts lowercases the extension, but this case-sensitive comparison sets gguf to false. The native loader then calls index_safe and treats the GGUF file as a SafeTensors directory, so a model that was successfully detected as qwen4_exp cannot load. Apply the same case-insensitive extension check here.
Useful? React with 👍 / 👎.
| pub fn max_concurrent_sequences(&self) -> u32 { | ||
| crate::engine::hybrid_scheduler::scheduler_max_num_seqs_for(4) as u32 |
There was a problem hiding this comment.
When MLX_SERVE_FORCE_SERIAL=1, the scheduler executes Qwen4 turns exclusively, but this method still reports up to four concurrent sequences. packages/server/src/registry.ts uses this value to select and size its counting semaphore, so several requests bypass the server's exclusive FIFO and no longer count against its waiting-queue limit even though they only queue serially in native code. Match the other hybrid models by returning 1 while forced-serial mode is active.
Useful? React with 👍 / 👎.
1 participant
Adds Qwen3.8-Flash-Next (
qwen4_exp) toloadModel()andloadSession()for Hugging Face directories and split GGUF checkpoints. Large checkpoints can generate with a bounded resident working set and SSD reads for cold weights; GGUF quantized codes are preserved during import.The MLX submodule advances to published commit
6908f3fd2for transposed K-quant expert gathers on NAX. Small reference fixtures, regeneration scripts, a guarded end-to-end smoke, and a public-API benchmark accompany the runtime.Validation on M5 Max / 128 GiB:
Paged inference currently requires macOS Metal. MTP is opt-in; audio, video and image-bearing MTP are unsupported. Full BF16 generation and real-model contexts beyond 2,048 tokens remain unvalidated. The documented 1,320 prefill tok/s result is from the pre-cleanup build and its specified workload; this PR does not claim 1,500 tok/s or a matched mlx.fast comparison.
Note
High Risk
Large new inference path touching scheduling, SSD weight residency, paged KV, and cancellation on a dedicated model thread; incorrect cache or slot lifetimes could affect stability or memory under load.
Overview
Adds Qwen3.8-Flash-Next (
qwen4_exp) end-to-end: a new SSD-oriented model family with bounded resident weights, GGUF/HF loading, optional auxiliary checkpoints for MTP/vision metadata, and publicQwen4ExpModelAPIs for paged chat sessions (sync/stream, tool turns), async cache reset, scheduler telemetry, and residency introspection.The Rust runtime introduces a full
qwen4_expstack—MoE with expert slotting and batched prefill, sparse/window attention with paged KV, linear (GDN) layers, PLE, native MTP draft paths, and memory admission tied to device headroom—plus loader/registry and README coverage as an inference-only SSD model type.The hybrid scheduler gains an opt-in
scheduler_wired_byteshook so scheduled turns can run under aWiredLimitContextduring generation.MxArray::deep_copyis fixed so sliced copies keep zero Metal buffer offset after source handles drop.Reviewed by Cursor Bugbot for commit 51bb619. Bugbot is set up for automated code reviews on this repo. Configure here.