I have implemented a little framework for structured output, chat and function calling. Can generate function calling grammars and structured output grammars:
https://github.com/Maximilian-Winter/llama-cpp-agent

@abetlen Could you tell me if this is useful for you? I can also integrate just the grammar generator into llama-cpp-python and make a pull request.

I am very interested in this. Will be reviewing when I have some time.

Be careful how you promote this chat handler. With this prompt it is only suitable for function calling, which might be confusing if you expect the handler to provide an OpenAI compatible experience with tool calls only when needed.

I am also not entirely sure if openhermes mistral was trained on these specific instructions or that it would work with other models as well.

If it just good prompt engineering to coerce a tool call, then langchain already has some tools to do so using reAct
https://github.com/williamcotton/empirical-philosophy/blob/main/articles/how-react-prompting-works.md

@vriesdemichael I get the concern but I've also seen very promising results for function calling with instruction tuned models. Additionally the goal here is also to have a skeleton for function calling that implements all of the nuances of the OpenAI API (multi-tool calling, legacy functions api, auto tool choice, streaming) so that we can adapt it to properly finetuned models.

Just missing automatic multi-tool use now.