Repository navigation
Cuda Dockerfile - model produces garbage output on GPU (if layers offloaded to GPU) #597
Description
Current Behavior
If we use the cuda dockerfile (https://github.com/abetlen/llama-cpp-python/blob/main/docker/cuda_simple/Dockerfile) and use it with any ggml model file and offloading layers to gpu, model outputs garbage text.
By garbage I mean, absolute garbage - symbols, special characters, Unicode characters etc, no English or any language.
Environment and Context
Running on centos 7 with docker and nvidia docker toolkit installed
Instance - aws g4dn.12xlarge (4 Nvidia T4 - 64GB VRAM)
Exact same issue seen in a repo related to Cublas in koboldcpp which also kind of uses llamacpp
issue: bartowski1182/koboldcpp-docker#1
fix commit: bartowski1182/koboldcpp-docker@331326a
Fix
I'll raise a pull request, with the updated dockerfile.
Cheers !🥂
Activity
pradhyumna85 commentedon Aug 15, 2023
@afafw, The only meaningful word in your output - "The" probably came due to not all layers offloaded to GPU(s), am I correct?
afafw commentedon Aug 15, 2023
Actually, that wasn't the case, I loaded all 35 layers into it. But I understand what you mean, since I did tried to load only part of network into gpu and it wasn't completely rubbish. It was just generating weird words.
afafw commentedon Aug 15, 2023
I think the key is also "ENV CUDA_DOCKER_ARCH=all" as I just saw your pull request.
pradhyumna85 commentedon Aug 15, 2023
@afafw, yeah I already fixed it, also raised a PR with the issue itself. You can check my PR, which is much shorter and cleaner.
Thank you anyway.
afafw commentedon Aug 15, 2023
Yeah, I just noticed it, I probably deleted this line
I saw it just after I send this, I thought I probably deleted this. I think your dockerfile probably works. As mine was simply building from source.
pradhyumna85 commentedon Aug 18, 2023
Fixed with the merge of PR #598