You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+16-2Lines changed: 16 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -48,15 +48,29 @@ For detailed installation instructions and requirements, see the [Installation G
48
48
49
49
### Quick Start
50
50
51
-
#### 1. Start an OpenAI Compatible Server (vLLM)
51
+
#### 1a. Start an OpenAI Compatible Server (vLLM)
52
52
53
53
GuideLLM requires an OpenAI-compatible server to run evaluations. [vLLM](https://github.com/vllm-project/vllm) is recommended for this purpose. To start a vLLM server with a Llama 3.1 8B quantized model, run the following command:
For more information on starting a vLLM server, see the [vLLM Documentation](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html).
59
+
#### 1b. Start an OpenAI Compatible Server (Hugging Face TGI)
60
+
61
+
GuideLLM requires an OpenAI-compatible server to run evaluations. [Text Generation Inference](https://github.com/huggingface/text-generation-inference) can be used here. To start a TGI server with a Llama 3.1 8B using docker, run the following command:
0 commit comments