@chraac great work! Could you also include the build flags in the README as well as maybe a small section on running the rpc servers?

@chraac great work! Could you also include the build flags in the README as well as maybe a small section on running the rpc servers?

yeah, sure, will add to readme, thanks for the reply!

@abetlen add a section in README.md about how to build the rpc backend.

@abetlen , could you have another look in convenient please? have added a section regarding how to build the RPC backend package.

Hi, I apologize for asking this here, but I'm trying to understand how to use the RPC_SERVERS functionality to enable inference across multiple machines. Can anyone help clarify this for me or give me an example?. Thanks in advance.

How do I use this...?

Hi @juanjfrancisco @statchamber , sorry for the inconvient here, currently the GGML_RPC flag is disabled by default at the upstream repo, so you maight need to install the python package from the source:

1. clone the repo: git clone https://github.com/abetlen/llama-cpp-python.git
2. For rpc client, which is the endpoint that make the rpc request, could be installed by following command into python env:
   CMAKE_ARGS='-DGGML_RPC=on -DGGML_OPENMP=on' python3 -m pip install --verbose -e .
3. For the rpc server that will actually execute the rpc request, could be installed by:

   cd vendor/llama.cpp
   mmkdir -p build_rpc_client
   pushd  build_rpc_client
   cmake -DGGML_RPC=ON -DGGML_CUDA=ON .. && cmake --build . --config Release
   popd