sampleFromLogits(topK = 0, topP < 1) is quadratic in the vocabulary (~10 s per token on Llama 3.2) #412

Summary

llm-core/.../SamplingUtils.kt sampleFromLogits(logits, temperature, topK, topP, random): when topK <= 0 and topP < 1f, k becomes logits.size and the insertion-into-k-sized-scratch selection loop degenerates to O(n²). On Llama-3.2-3B (vocab 128 256) that is ~1.6·10¹⁰ compares per token — measured ~10 s per generated token with generateUntilStop(topK = 0, topP = 0.9f), versus ~0.3 s with topK = 40.

generateUntilStop routes any topP < 1f through this overload (filtered = topK > 0 || topP < 1f), so a caller asking for nucleus sampling alone silently falls into the trap. -Dskainet.strict.kernels=true does not catch it (no kernel is involved).

Repro

Daily-StandAPP LlamaGenerateStage before the workaround: 39-token prompt, 5 generated tokens in 57 s; jstack samples all in SamplingUtilsKt.sampleFromLogits(SamplingUtils.kt:122).

Suggested fix

• When topK <= 0 and topP < 1, select with a partial sort / heap over a bounded candidate set (e.g. a quickselect on the probability mass), or default topK to a sane value (llama.cpp uses 40) when only top-p is requested.
• Document the interaction on generateUntilStop.

Workaround

Pair top-p with a finite top-k (Daily-StandAPP now passes topK = 40 whenever topP < 1).