I found this cool project doesn't support baichuan2, so I make this PR to support it.

What is added

I added a new chat format named "baichuan-2" to support baichuan2 chat model.
I only tested Baichuan2-7B-Chat. Not sure whether Baichuan2-13B-Chat can work.

How to use it

1. convert baichuan2 mode to baichuan1

see: https://github.com/baichuan-inc/Baichuan2

replace the lm_head.weight with new one.

2. uses llama.cpp's convert.py to convert Baichuan2-7B-Chat to GGUF format.

python3 convert.py --outfile models/baichuan-2-7b-chat-baichuan-f16.gguf ../baichuan/Baichuan2-7B-Chat-Baichuan/

3. quantize the model, take Q3_K_M for example.

./quantize models/baichuan-2-7b-chat-baichuan-f16.gguf models/baichuan-2-7b-chat-baichuan-Q3_K_M.gguf Q3_K_M

4. launch llama-cpp-python server and test. remember to use " --chat_format baichuan-2"

python3 -m llama_cpp.server --model models/baichuan-2-7b-chat-baichuan-Q3_K_M.gguf --host 0.0.0.0 --port 9000 --n_gpu_layers -1 --chat_format baichuan-2