llama-cpp-python 0.4.0 wheel with the adaptive KV-streaming llama.cpp build.

Build provenance

• Superproject: LLAI-lab/llama-cpp-python main (0b369c7)
• vendor/llama.cpp: kv-stream lineage 22abd08
  = RaymondHuang210129/llama.cpp-adaptive-kv-stream feature/adaptive-kv-stream (4ba0b9b)
  • clip.cpp graceful mmproj OOM fix (throw instead of aborting)
• Flags: GGML_CUDA=ON, GGML_CUDA_FA_ALL_QUANTS=ON, sm_120a, MMQ forced

Requirements

• Windows x64, Python 3.13 (cp313)
• NVIDIA RTX 50 series GPU (sm_120a only — other architectures are not compiled in)
• CUDA 13.0 toolkit runtime DLLs (cudart64_13, cublas64_13, cublasLt64_13
  resolved from CUDA\v13.0\bin\x64 on PATH; not bundled)
• libomp140.x86_64.dll is bundled

Notes

• llama_adapter_lora_init_from_file_ptr (JamePeng llama.cpp extension) is
  declared optional and unavailable in this build; imports are unaffected.
• Build recipe: build-wheel-file.bat in the repo root.

Rebuild of the 1kvstream wheel with:

• merged JamePeng upstream (MTMDBaseHandler refactor, sampler binding fix)
• graceful error instead of hard abort when mmproj weight buffer allocation fails (CUDA OOM when the GPU is shared with ComfyUI etc.)
• vendored llama.cpp pinned at the kv-stream lineage (d873e5d + fix, mirrored at LLAI-lab/llama.cpp branch feature/adaptive-kv-stream)

Build: MSVC 14.44 + CUDA 13.0, GGML_CUDA=ON, sm_120a, GGML_CUDA_FA_ALL_QUANTS=ON. cublas64_13.dll / cublasLt64_13.dll are not bundled; they resolve from the CUDA toolkit at runtime.

Prebuilt wheel of llama-cpp-python 0.3.49 with adaptive KV cache streaming.

• Vendored llama.cpp: RaymondHuang210129/llama.cpp-adaptive-kv-streaming @ d873e5db9 (feature/adaptive-kv-stream) — block-granular KV streaming for CUDA, exact attention over full context
• Adaptation patches: see commit 66f0df6 (kv_stream_stage_mib param, optional mtmd bindings, struct layout fixes)

Usage:

from llama_cpp import Llama
llm = Llama(model_path=..., n_gpu_layers=-1, kv_stream_stage_mib=512)

or set env LLAMA_ARG_KV_STREAM_STAGE_MIB=512.

Requirements:

• Windows x64, Python 3.13 (cp313)
• NVIDIA RTX 50 series (kernels compiled for sm_120a only), driver supporting CUDA 13
• cuBLAS 13 runtime DLLs (cublas64_13.dll / cublasLt64_13.dll) on PATH — not bundled (e.g. CUDA Toolkit bind)
• CPU backend compiled natively on the build machine; GPU path recommended

构建说明（中文）：基于 kv-stream fork 的 llama.cpp 编译，附加参数支持 -fa on -ctk q8_0 -ctv q4_0 --kv-stream-stage-mib N；上下文可开 80K+ 而不溢出显存。验证环境：RTX 5070 Ti 16GB + Qwen3.8-27B IQ4_XS。