Skip to content

feat(model): complete Qwen3.8 support - #5339

Merged
qinxuye merged 11 commits into
xorbitsai:mainfrom
qinxuye:feat/qwen3.8-support
Aug 17, 2026
Merged

feat(model): complete Qwen3.8 support#5339
qinxuye merged 11 commits into
xorbitsai:mainfrom
qinxuye:feat/qwen3.8-support

Conversation

@qinxuye

@qinxuye qinxuye commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • extend Qwen3.8-27B with BF16 GGUF and MLX 4-bit, 8-bit, and BF16 variants, and add the Qwen3.8-2.4T-A95B model as qwen3.8-max
  • wire the required MLX, llama.cpp, vLLM, and SGLang architecture/version support and document the new model and engine requirements
  • fix Qwen tool-call streaming for multiple calls, propagate model sampling defaults and top_k through MLX, and preserve final and usage chunks
  • make automatic model-hub selection avoid Hugging Face endpoints that would use environment proxy traffic, while honoring HF_ENDPOINT, NO_PROXY, and explicit per-launch or service-level source selections

Tests

  • pytest -q (13 focused Qwen3.8, MLX, tool streaming, and model-source regression tests)
  • pre-commit run --files (all 18 changed files)

@XprobeBot XprobeBot added this to the v3.x milestone Aug 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the qwen3.8-max model, updates automatic download hub detection to avoid routing large downloads through environment-configured proxies, and adds top_k sampling support to the MLX engine. It also implements streaming tool-call parsing for MLX and adds comprehensive tests. The reviewer feedback suggests filtering out None values when updating the model generation configuration in MLX to prevent downstream type errors, and casting top_k to an integer to avoid cache key mismatches or type errors in the sampler.

Comment thread xinference/model/llm/mlx/core.py
Comment thread xinference/model/llm/mlx/core.py Outdated

@OliverBryant OliverBryant left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one streaming correctness issue.

Comment thread xinference/model/llm/tool_parsers/qwen_tool_parser.py Outdated

@OliverBryant OliverBryant left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one ordering issue in the follow-up fix.

Comment thread xinference/model/llm/tool_parsers/qwen_tool_parser.py Outdated

@OliverBryant OliverBryant left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one remaining multi-call ordering issue.

Comment thread xinference/model/llm/tool_parsers/qwen_tool_parser.py Outdated

@OliverBryant OliverBryant left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the three streaming-order fixes from the earlier rounds hold, and re-checked the two Gemini findings as genuinely fixed. Three new items below: one real streaming regression, one connectivity regression, one doc/impl mismatch.

Comment thread xinference/model/llm/mlx/core.py Outdated
Comment thread xinference/model/llm/tool_parsers/qwen_tool_parser.py Outdated
Comment thread xinference/model/utils.py Outdated
Comment thread doc/source/models/sources/sources.rst Outdated
@OliverBryant

Copy link
Copy Markdown
Collaborator

Re-checked all four fixes in b026812d2 against the new head — all confirmed, no regressions. LGTM.

P1 MLX cache aliasinglist(cached_tokens) matches upstream's own cache_key[:] at mlx_lm/server.py:874. The test is a real one: it keeps the generator-owned list, appends to it after insertion, and asserts the stored key is a distinct object.

P2 malformed sibling call — re-ran my original repro against both revisions:

BAD then GOOD, same delta:     main: RAISED KeyError   ->  now: (None, 'good', {'x': 1}, 1)
BAD first, GOOD later delta:   main: RAISED KeyError   ->  now: (None, 'good', {'x': 1}, 1)

Now strictly better than main, which still raises in both. I also checked index bookkeeping after a skip: the malformed call keeps its slot, so with good / malformed / good the survivors emit as index 0 and 2 with stable call IDs and a single name emission each. Non-contiguous indexes are correct here — index identifies the call, and the alternative would renumber calls the client may already have seen. The partial-name path is regex-only, so it can't raise.

P2 CA bundle — verified the narrowed fix does both things it needs to:

HTTPS_PROXY + ALL_PROXY + REQUESTS_CA_BUNDLE set, trust_env=True,
proxies={"http": None, "https": None, "all": None}
  -> selected proxy: None
  -> verify: /nonexistent/ca.pem    (CA bundle preserved)

P3 docs — scheme-dependent wording is accurate now, and the code comment at model/utils.py:663 was updated to match.

All three new regression tests fail on b0f47155d and pass on b026812d2, so they're genuinely covering the fixes. Also re-ran the earlier streaming-order cases (prefix content, inter-call content, close/content/open, complete-following-call) — all still correct.

One thing I did not verify: no MLX hardware here, so the prompt-cache path is only checked against upstream mlx-lm's logic and the unit tests, not a live batch run.

@OliverBryant OliverBryant left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qinxuye
qinxuye merged commit a9cfdae into xorbitsai:main Aug 17, 2026
15 checks passed
@qinxuye
qinxuye deleted the feat/qwen3.8-support branch August 17, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants