Skip to content

tests

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

tests

README.md

Tests

Running

From the repository root, with the compiled package installed:

python -m pytest
python -m pytest tests/test_state.py

Tests use local model files and never download them. Unconfigured model tests skip locally; the core Transformer, Hybrid and vision paths are required in Actions. Configured paths must exist and load successfully. Optional embedding and DFlash model pairs skip when unset.

Models

Environment variable Model / purpose
LLAMA_TEST_TRANSFORMER_MODEL Qwen2.5-0.5B-Instruct Q4_K_M; Transformer runtime, KV state and grammar
LLAMA_TEST_HYBRID_MODEL Qwen3.5 with MTP weights; Hybrid checkpoints and speculative decoding
LLAMA_TEST_MMPROJ Matching Qwen3.5 projector; vision and MTP image tests
LLAMA_TEST_MULTIMODAL_EMBEDDING_MODEL / LLAMA_TEST_MULTIMODAL_EMBEDDING_MMPROJ EmbeddingGemma 2 model and projector
LLAMA_TEST_QWEN3VL_EMBEDDING_MODEL / LLAMA_TEST_QWEN3VL_EMBEDDING_MMPROJ Qwen3-VL-Embedding model and projector
LLAMA_TEST_DFLASH_MODEL / LLAMA_TEST_DFLASH_DRAFT DFlash target and draft model
LLAMA_TEST_DFLASH2_MODEL / LLAMA_TEST_DFLASH2_DRAFT DFlash2 target and draft model

CI pins Qwen2.5 and Qwen3.5-0.8B with its projector. Larger compatible local models can use LLAMA_TEST_N_GPU_LAYERS=-1 for full GPU offload; the default is 0. Tokenizer tests use the vocabulary-only GGUF in vendor/llama.cpp/models.

Coverage

Module Main checks
test_cache.py RAM/disk/trie prefix matching, thresholds, ties, eviction and memory bounds
test_state.py Snapshot ownership, Hybrid restore, zero checkpoints, cache reuse and failure recovery
test_speculation.py MTP/DFlash verification, rollback, interrupted continuation and score callbacks
test_runtime.py Native API and batch parity, resource lifetimes, sampling, KV state and embeddings
test_media.py Vision/audio handling, MTP image requests and partial-decode recovery
test_formats.py Chat formatting and grammar constraints
test_rpc.py RPC backend, endpoint validation and device ownership

Share model weights and owned cold references, reset live contexts between cases, and isolate patches and disk caches. Aggregate equivalent inputs while preserving boundary, failure-recovery and ownership assertions.

Quantized model comparisons

With quantized GGUFs such as Q4_K_M, full-prompt and prefix/suffix decoding can produce different logits because their batch shapes differ. Close scores can swap order and change greedy output even at temperature=0 with a fixed seed. Results can also vary by backend; different text alone does not establish incorrect cache reuse.

Compare reuse against an independent native context with the same model, runtime settings, batch boundaries and output flags. Keep assertions on logits, evaluated suffixes, token histories and native positions to detect actual state errors.

Benchmarks

See the scripts' --help for model, baseline and result options:

python -m examples.benchmark.benchmark_trie_cache --help
python -m examples.benchmark.benchmark_default_reuse --help

The first compares Trie maintenance and cache reuse. The second measures the default cache=None prefix reuse and MTP rollback path. Both support comparison against saved baseline sources and check generated tokens against cold references.