Summary

v0.4.1 with experimental PrismML fork used

Includes breaking features, like 'mtmd_helper_init_opt'/video_override options.

So do not use production, just experiment only.

CUDA Toolkit 13.2 might be required to run.

Personal experimental build which tries to use official mtp/dflash/dspark implementation.

Sync v0.3.47 from upstream repo.

Full Changelog: v0.3.46-native-speculative.4...v0.3.47-native-speculative.2

Just simple issue fix which part does not exists in parameters.

Please read full changes in 0.2 version.

Just simple issue fix which Llama.reset() does not reset context memory.

Please read full changes in previous version.

This release packages an experimental llama-cpp-python bridge to llama.cpp's
native speculative decoding implementations. Build 2 extends the previous
DFlash/DSpark work with external Gemma 4 MTP and embedded/internal Qwen 3.5 MTP.

The primary goal remains practical local use with ComfyUI: load an LLM, generate
one text response, collect speculative statistics, and unload it before a larger
image/video model is loaded.

Build information

• Base package: llama-cpp-python 0.3.46
• Source tag: v0.3.46-native-speculative.2
• Source commit: dce46922976684a85d2114bd435d30ba3be5ed7f
• Platform: Windows x64
• CUDA Toolkit used for build: CUDA 13.2
• CUDA architecture used for validation: SM 8.6
• Validated GPU: NVIDIA GeForce RTX 3090 Ti
• Fully validated environment: CPython 3.13 / ComfyUI / CUDA
  ComfyUI and real-model validation matrix

Experimental features

Native DFlash and DSpark

• External DFlash/DSpark GGUF provider through llama.cpp common/speculative
• Text generation
• Request/lifetime draft and acceptance statistics
• Native stop/EOG completion propagation
• Text-only clear-and-reprefill fallback on context exhaustion
• Experimental initial MTMD/mmproj embedding mirroring

Validated combinations include:

• Muse-Glimmer 30B + matching DFlash GGUF, text
• Muse-Glimmer 30B + mmproj + matching DFlash GGUF, single-shot vision
• Qwen 3.5 9B + matching DFlash GGUF, text

Muse-Glimmer is the only DFlash vision combination validated successfully in
this build. Qwen 3.5/3.6 DFlash vision can fail because the target and draft
contexts do not retain consecutive multimodal sequence positions; treat it as
unsupported.

Gemma 4 external MTP

• gemma4 target GGUF
• Separate matching gemma4-assistant MTP GGUF
• Architecture, hidden-width, vocabulary, NextN-layer, and shared-target-context
  validation
• Text-only generation
• Request-local MTP statistics

The assistant must match the target lineage. Architecture compatibility alone
does not guarantee useful draft acceptance. Fine-tuned, uncensored, abliterated,
or otherwise modified targets may reject nearly every draft from an assistant
trained for the original model.

Qwen 3.5 embedded/internal MTP

• No separate assistant GGUF
• Uses NextN/MTP layers embedded in the target GGUF
• Automatically enables target MTP tensor loading
• Validates that the target contains usable NextN layers
• Independent/shared MTP memory lifecycle handling
• Clears stale recurrent hidden state between independent requests
• Text-only generation

Validated model:

Qwen3.5-9B-UD-Q4_K_XL.gguf
embedded NextN layers: 1

Local greedy CUDA validation showed deterministic repeated responses, normal
EOG termination, active draft statistics, and a substantial request-time
improvement. Results are hardware-, model-, quantization-, prompt-, and
configuration-dependent.

Installation

Install the wheel matching the exact Python ABI used by ComfyUI or the target
virtual environment.

CPython 3.13:

python -m pip install --force-reinstall --no-deps `
    .\llama_cpp_python-0.3.46-cp313-cp313-win_amd64.whl

The wheel filename does not encode its CUDA Toolkit or GPU architecture. Read
the build information above before installing it.

Verify the installed package and CUDA backend:

python -c "import llama_cpp; print(llama_cpp.__file__); print(llama_cpp.llama_cpp.llama_supports_gpu_offload())"

Verify the experimental API:

python -c "from llama_cpp.llama_speculative import LlamaNativeSpeculativeDecoding; print(LlamaNativeSpeculativeDecoding)"

The class is intentionally not exported from the top-level llama_cpp
namespace.

Minimal API examples

DFlash or DSpark with an external draft GGUF

from llama_cpp import Llama
from llama_cpp.llama_speculative import LlamaNativeSpeculativeDecoding

draft = LlamaNativeSpeculativeDecoding(
    model_path="path/to/matching-dflash.gguf",
    spec_type="draft-dflash",
    n_gpu_layers="all",
    n_max=8,
)

llm = Llama(
    model_path="path/to/target.gguf",
    draft_model=draft,
    n_gpu_layers="all",
    n_ctx=4096,
    n_batch=512,
    n_ubatch=512,
)

Use spec_type="draft-dspark" for a compatible DSpark artifact.

Gemma 4 with an external MTP assistant

draft = LlamaNativeSpeculativeDecoding(
    model_path="path/to/matching-gemma4-assistant.gguf",
    spec_type="draft-mtp",
    n_gpu_layers="all",
    n_max=2,
)

llm = Llama(
    model_path="path/to/gemma4-target.gguf",
    draft_model=draft,
    n_gpu_layers="all",
    n_ctx=4096,
    native_context_reprefill=False,
)

Qwen 3.5 with embedded/internal MTP

draft = LlamaNativeSpeculativeDecoding(
    model_path=None,
    spec_type="draft-mtp",
    n_gpu_layers="all",
    n_max=2,
)

llm = Llama(
    model_path="path/to/qwen35-with-embedded-nextn.gguf",
    draft_model=draft,
    n_gpu_layers="all",
    n_ctx=4096,
    native_context_reprefill=False,
)

print("NextN layers:", llm.n_layer_nextn())

Copy statistics before closing the model:

stats = dict(draft.stats)
print(stats)
llm.close()

Relevant fields:

• draft_calls
• accept_calls
• drafted_tokens
• accepted_tokens
• acceptance_rate
• mean_accepted_tokens

Draft activity proves that the provider was called; it does not by itself prove
a speedup. Very low acceptance can make speculative generation slower than the
target-only path.

Validation performed

• Focused final native speculative regression: 60 passed
• Full Python regression recorded for the internal-MTP implementation: 97 passed
• Fresh CUDA DLL build and runtime ABI check
• Clean virtual-environment wheel installation
• ComfyUI CPython 3.13 installation and real workflow execution
• CUDA backend and all-layer GPU offload detection
• Muse-Glimmer DFlash text and vision generation
• Gemma 4 external MTP text generation
• Qwen 3.5 embedded/internal MTP text generation
• Correct finish_reason handling for stop, EOG, and length cases
• Same-instance reset and second-request lifecycle checks
• Draft/acceptance statistics and deterministic validator checks

The validation is intentionally focused on the experimental single-request use
case. It is not a claim of production completeness.

Known limitations

• Experimental, unofficial, and unsupported
• Single sequence only (n_seq_max=1)
• No continuous batching
• MTP is text-only
• No MTP multimodal embedding support
• No MTP context shifting or clear-and-reprefill fallback
• No MTP prefix reuse
• No native grammar or JSON Schema integration
• No native custom logits processor integration
• No Python state-cache serialization/restoration
• DFlash/DSpark multimodal support is limited to initial single-shot prefill
• Qwen DFlash vision is currently unsupported
• Streaming, tool-call combinations, and broad chat-handler compatibility have
  not received the same validation coverage
• Target-only and speculative generation are not guaranteed to be token-exact
  because batched GPU verification can introduce deterministic numerical
  divergence near close logits
• A compatible architecture and vocabulary do not guarantee useful acceptance;
  the draft/provider must match the target model lineage
• High additional VRAM use may occur when target, external draft, and mmproj
  components coexist
• CUDA architectures other than SM 8.6, other GPUs, operating systems, and
  non-CUDA backends have not been validated

Use target-only fallback when unsupported features are required or when measured
acceptance is too low to provide a speed benefit.

Wheel integrity

CPython 3.13

File: llama_cpp_python-0.3.46-cp313-cp313-win_amd64.whl
SHA-256: 8AFA472D357CA384FE89C04B2EAF3FD6F4BD1B72C0828CC43B67869CBC80FC78

Development notes archive

If a separate development-notes ZIP is attached, it contains raw Korean notes,
local test scripts, prompts, implementation plans, diagnostic reports, and
regression results.

The archive is historical and diagnostic material only. It is not installed by
the wheel and is not supported public documentation. It may preserve
machine-specific Windows paths and stale intermediate conclusions. Build trees,
model files, credentials, and authentication tokens should not be included.

Project status

This release is a frozen experimental reference snapshot rather than a promise
of a stable public API. Future versions may migrate to an official high-level
MTP/speculative API if one is completed upstream. Existing ComfyUI integrations
should keep provider creation behind a small compatibility adapter.

Experimental Native Speculative Decoding 0.1

This pre-release contains a locally built llama-cpp-python wheel with an
experimental bridge to llama.cpp native speculative decoding.

Build information

• Base package: llama-cpp-python 0.3.46
• Source tag: v0.3.46-native-speculative.1
• Source commit: d76ef31af0266f76a520b3370095a0d556bd6bc2
• Platform: Windows x64
• Python ABI: CPython 3.13
• CUDA Toolkit used for build: CUDA 13.2
• CUDA architecture: SM 8.6
• Validated GPU: NVIDIA GeForce RTX 3090 Ti
• Wheel: llama_cpp_python-0.3.46-speculative-cp313-cu132-win_amd64.whl

Experimental features

• Native DFlash and DSpark speculative decoding bridge
• Muse-Glimmer DFlash GGUF validation
• Text generation
• Experimental MTMD/mmproj multimodal prefill
• Text context-exhaustion clear-and-reprefill fallback
• Native stop/EOG completion propagation
• Native speculative acceptance statistics

Validation performed

The wheel was installed into clean and ComfyUI Python 3.13 environments.

Validated workloads:

• CUDA backend loading and GPU offload detection
• Muse-Glimmer 30B target model with DFlash draft model
• Text chat completion
• Image input through MTMD/mmproj
• Correct finish_reason=stop
• Native speculative draft and acceptance statistics
• ComfyUI virtual-environment compatibility

Text workloads showed substantial acceleration in local testing.

The short vision smoke test completed correctly, but did not show a meaningful
performance improvement because draft acceptance was low and image prefill is
not accelerated by speculative decoding.

Known limitations

• Experimental and unsupported
• Single sequence only (n_seq_max=1)
• No continuous batching
• No native grammar integration
• No custom logits processor integration
• No Python state-cache restoration
• General context shifting is not supported
• Multimodal support is intended for initial single-shot prefill
• Target-only and speculative generation are not guaranteed to be token-exact
• High additional VRAM use is expected because target, draft, and mmproj
  components may coexist
• Other CUDA architectures, GPUs, Python versions, operating systems, and
  compute backends have not been tested
• A CUDA-host compute-buffer size warning may appear during teardown; no
  associated generation failure was observed in local testing

Installation

Use only with CPython 3.13 on a compatible Windows x64 CUDA environment.

python -m pip install --force-reinstall --no-deps `
    .\llama_cpp_python-0.3.46-speculative-cp313-cu132-win_amd64.whl

Verify CUDA support:

python -c "import llama_cpp; print(llama_cpp.__file__); print(llama_cpp.llama_cpp.llama_supports_gpu_offload())"

Import the experimental API with:

from llama_cpp.llama_speculative import LlamaNativeSpeculativeDecoding

The class is intentionally not exported from the top-level llama_cpp
namespace.

Integrity

Source commit:

d76ef31

Wheel SHA-256:

E8D804A00AB5767435F867558C2305AEC221F02D11203DF6985986E4D60C140B

Development notes archive

native-speculative-development-notes-v0.1.zip contains raw Korean development
notes, local test scripts, prompts, diagnostic reports, and regression results.

The archive is provided for historical and diagnostic reference only. It is not
part of the installed package or supported public documentation. It
intentionally preserves machine-specific Windows paths and may contain stale
intermediate conclusions. No known credentials or authentication tokens were
found during automated and manual review.