Skip to content

GTX 1060 CUDA not working for Qwen3-4B-Q4_K_M.gguf with llama-cpp-python 0.3.23-cu125 on Windows #2234

Closed

Description

@yc2768

Title:
GTX 1060 (sm_61) Windows 10: ggml-cuda.cu:102 CUDA error with Qwen3-4B-Q4_K_M.gguf

Body:
Environment:

  • OS: Windows 10 64-bit
  • GPU: NVIDIA GeForce GTX 1060 6GB (compute capability 6.1)
  • Driver: 53x.xx (supports CUDA 12.5)
  • llama-cpp-python: 0.3.23-cu125 (prebuilt wheel)
  • Model: Qwen3-4B-Q4_K_M.gguf
  • Python: 3.10

Error messages:

  1. llama_context: n_ctx_seq (512) < n_ctx_train (40960)
  2. ggml-cuda.cu:102: CUDA error

Symptoms:

  • With default n_ctx=512 and n_gpu_layers=35, CUDA error occurs immediately.
  • After changing n_ctx=2048 and n_gpu_layers=20, the warning remains, and CUDA error still happens.
  • GPU usage jumps briefly then drops to 0%; model falls back to CPU.

Suspected cause:
The prebuilt cu125 wheel does NOT include sm_61 (Pascal/GTX 1060) kernels, leading to CUDA error when offloading layers.

Questions:

  1. Does the official cu125 wheel support sm_61?
  2. Is there a Windows 10 prebuilt wheel with sm_61 included?
  3. What is the recommended n_ctx / n_gpu_layers for Qwen3-4B-Q4_K_M.gguf on GTX 1060 6GB?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions