Local llama-cpp-provider for gptscript

Lets first write models.json, where we can specify which models we want to use.

{
	"functionary": {
		"repo_id": "meetkai/functionary-small-v2.2-GGUF",
		"filename": "functionary-small-v2.2.q4_0.gguf",
		"chat_format": "functionary-v2"
	},
	"functionary-7b": {
		"repo_id": "meetkai/functionary-7b-v2.1-GGUF",
		"filename": "functionary-7b-v2.1.q4_0.gguf",
		"chat_format": "functionary-v2"
	}
}

We can now set up llama-cpp-provider with:

git clone https://github.com/mircodz/llama-cpp-provider
cd llama-cpp-provider
python -m venv .venv
source ./.venv/bin/activate
pip install --upgrade -r requirements.txt

And run it as follows:

LLAMA_SPEC_PATH=models.json ./run.sh

To test it out, let’s run the following command:

gptscript --default-model="functionary-7b from http://127.0.0.1:8000/v1" https://get.gptscript.ai/echo.gpt 'Hello, World!'