Skip to content

3.model-cache-quick-start.md

Latest commit

 

History

History
97 lines (72 loc) · 2.99 KB

3.model-cache-quick-start.md

File metadata and controls

97 lines (72 loc) · 2.99 KB
Raw

Quick start

This topic describes how to set up and use ModelCache.

You can find the start script in flask4modelcache.py and flask4modelcache_demo.py.

  • flask4modelcache_demo.py: A quick test service that embeds SQLite and FAISS. No database configuration required.
  • flask4modelcache.py: The standard service that requires MySQL and Milvus configuration.

Dependencies

  • Python: V3.8 or above

  • Package installation

    pip install -r requirements.txt

Start service

Start demo

  1. Download the embedding model bin file from Hugging Face. Place it in the model/text2vec-base-chinese folder.
  2. Start the backend service:
cd CodeFuse-ModelCache
python flask4modelcache_demo.py

Start standard service

Before you start standard service, do these steps:

  1. Install MySQL and import the SQL file from reference_doc/create_table.sql.

  2. Install vector database Milvus.

  3. Configure database access in:

    • modelcache/config/milvus_config.ini
    • modelcache/config/mysql_config.ini

  4. Download the embedding model bin file from Hugging Face. Put it in model/text2vec-base-chinese.

  5. Start the backend service:

    python flask4modelcache.py

Visit the service

The service provides three core RESTful API functionalities: Cache-Writing, Cache-Querying, and Cache-Clearing.

Write cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'insert'
scope = {"model": "CODEGPT-1008"}
chat_info = [{"query": [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "你是谁?"}],
                  "answer": "Hello, I am an intelligent assistant. How can I assist you?"}]
data = {'type': type, 'scope': scope, 'chat_info': chat_info}
headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))

Query cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'query'
scope = {"model": "CODEGPT-1008"}
query = [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "Who are you?"}]
data = {'type': type, 'scope': scope, 'query': query}

headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))

Clear cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'remove'
scope = {"model": "CODEGPT-1008"}
remove_type = 'truncate_by_model'
data = {'type': type, 'scope': scope, 'remove_type': remove_type}

headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))