Skip to content

GGUF (Breaking Change to Model Files)#633

Merged
abetlen merged 11 commits intomainfrom
gguf
Aug 25, 2023
Merged

GGUF (Breaking Change to Model Files)#633
abetlen merged 11 commits intomainfrom
gguf

Conversation

@abetlen
Copy link
Copy Markdown
Owner

@abetlen abetlen commented Aug 24, 2023 •
edited
Loading

GGUF support for llama.cpp Closes #628

Currently works to update your old ggml v3 llama models run

python3 vendor/llama.cpp/convert-llama-ggmlv3-to-gguf.py --input <path-to-ggml> --output <path-to-gguf>

TODO

  • Fix tests
  • Move convert script into package to make it easier for people to migrate Add docs link to conversion script in llama.cpp
  • Fix detokenization bug (getting extra leading space)

@abetlen abetlen changed the title GGUF GGUF (Breaking Change) Aug 24, 2023
@abetlen abetlen changed the title GGUF (Breaking Change) GGUF (Breaking Change to Model Files) Aug 24, 2023
@rlancemartin rlancemartin mentioned this pull request Aug 25, 2023
Closed
@Ph0rk0z Ph0rk0z mentioned this pull request Aug 25, 2023
Closed
@abetlen abetlen merged commit 915bbea into main Aug 25, 2023
@sndani
Copy link
Copy Markdown

sndani commented Aug 26, 2023

Hello, excited about trying this out with the CodeLlama gguf model.

Followed the MacOS (Sonoma beta) instructions. How do I get the 'llama' shared library?

llama-cpp-python % python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1
Traceback (most recent call last):
....
File "... /llama-cpp-python/llama_cpp/llama_cpp.py", line 80, in
_lib = _load_shared_library(_lib_base_name)
File " ... /llama-cpp-python/llama_cpp/llama_cpp.py", line 71, in _load_shared_library
raise FileNotFoundError(
FileNotFoundError: Shared library with base name 'llama' not found

Thanks!

@abetlen
Copy link
Copy Markdown
Owner Author

abetlen commented Aug 26, 2023

@sndani try reinstalling with the --verbose flag, it's likely a build error and will be reported by cmake. If the issue isn't resolvable please open an issue and I'll take a look. Cheers

@sndani
Copy link
Copy Markdown

sndani commented Aug 26, 2023

@abetlen thanks for the great work and thanks for responding.
Yes, did run
% CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir --verbose

Turns out, cmake isn't getting to target libllama.so under vendor/llama.cpp (but 'make clean' try deleting it). This is a dev fix but will open an issue (or the next person who encounters this can) if this isn't only my environment for some reason.

% make clean
% cd vendor/llama.cpp
% LLAMA_METAL=on make libllama.so
% cd ../..
% pip install 'llama-cpp-python[server]'
% export LLAMA_CPP_LIB=./vendor/llama.cpp/libllama.so
% python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1

@reddiamond1234
Copy link
Copy Markdown

reddiamond1234 commented Sep 1, 2023

My model is now a lot slower... is there any solution to fix this?

@phronmophobic phronmophobic mentioned this pull request Sep 24, 2023
Closed
@rlancemartin rlancemartin mentioned this pull request Feb 13, 2024
Closed
@abdullrahmanesmael07
Copy link
Copy Markdown

abdullrahmanesmael07 commented Oct 17, 2024

Hi, i always get this error.

C:\Users\abdul\OneDrive\Desktop\Coding playground\python\LLM\llama-cpp-python>python -m pip install -e .
Obtaining file:///C:/Users/abdul/OneDrive/Desktop/Coding%20playground/python/LLM/llama-cpp-python
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (4.12.2)
Requirement already satisfied: numpy>=1.20.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (1.26.4)
Requirement already satisfied: diskcache>=5.6.1 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (5.6.3)
Requirement already satisfied: jinja2>=2.11.3 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from llama_cpp_python==0.3.1) (3.1.3)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\abdul\appdata\local\programs\python\python311\lib\site-packages (from jinja2>=2.11.3->llama_cpp_python==0.3.1) (2.1.5)
Building wheels for collected packages: llama_cpp_python
  Building editable for llama_cpp_python (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for llama_cpp_python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      *** scikit-build-core 0.10.7 using CMake 3.29.0 (editable)
      *** Configuring CMake...
      2024-10-17 10:21:25,535 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None
      loading initial cache file C:\Users\abdul\AppData\Local\Temp\tmp9my5wqdi\build\CMakeInit.txt
      -- Building for: NMake Makefiles
      CMake Error at CMakeLists.txt:3 (project):
        Running

         'nmake' '-?'

        failed with:

         no such file or directory


      CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
      CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
      -- Configuring incomplete, errors occurred!

      *** CMake configuration failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for llama_cpp_python
Failed to build llama_cpp_python
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (llama_cpp_python)

@cccampos123
Copy link
Copy Markdown

cccampos123 commented Jan 14, 2025

Hi abdullrahmanesmael07, if you are using windows, install visual studio community with c++ options and reboot. It will works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

GGUF Support

5 participants

@abetlen @sndani @reddiamond1234 @abdullrahmanesmael07 @cccampos123