Skip to content

Commit ea1431b

Browse files
authored
docs : add "Quick start" section for new users (#13862)
* docs : add "Quick start" section for non-technical users * rm flox * Update README.md
1 parent 71e74a3 commit ea1431b

File tree

3 files changed

+54
-27
lines changed

3 files changed

+54
-27
lines changed

README.md

Lines changed: 30 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,30 @@ Inference of Meta's [LLaMA](https://arxiv.org/abs/2302.13971) model (and others)
2828

2929
----
3030

31+
## Quick start
32+
33+
Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:
34+
35+
- Install `llama.cpp` using [brew, nix or winget](docs/install.md)
36+
- Run with Docker - see our [Docker documentation](docs/docker.md)
37+
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
38+
- Build from source by cloning this repository - check out [our build guide](docs/build.md)
39+
40+
Once installed, you'll need a model to work with. Head to the [Obtaining and quantizing models](#obtaining-and-quantizing-models) section to learn more.
41+
42+
Example command:
43+
44+
```sh
45+
# Use a local model file
46+
llama-cli -m my_model.gguf
47+
48+
# Or download and run a model directly from Hugging Face
49+
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
50+
51+
# Launch OpenAI-compatible API server
52+
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
53+
```
54+
3155
## Description
3256

3357
The main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide
@@ -230,6 +254,7 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo
230254

231255
</details>
232256

257+
233258
## Supported backends
234259

235260
| Backend | Target devices |
@@ -246,24 +271,18 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo
246271
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
247272
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
248273

249-
## Building the project
250-
251-
The main product of this project is the `llama` library. Its C-style interface can be found in [include/llama.h](include/llama.h).
252-
The project also includes many example programs and tools using the `llama` library. The examples range from simple, minimal code snippets to sophisticated sub-projects such as an OpenAI-compatible HTTP server. Possible methods for obtaining the binaries:
253-
254-
- Clone this repository and build locally, see [how to build](docs/build.md)
255-
- On MacOS or Linux, install `llama.cpp` via [brew, flox or nix](docs/install.md)
256-
- Use a Docker image, see [documentation for Docker](docs/docker.md)
257-
- Download pre-built binaries from [releases](https://github.com/ggml-org/llama.cpp/releases)
258-
259274
## Obtaining and quantizing models
260275

261276
The [Hugging Face](https://huggingface.co) platform hosts a [number of LLMs](https://huggingface.co/models?library=gguf&sort=trending) compatible with `llama.cpp`:
262277

263278
- [Trending](https://huggingface.co/models?library=gguf&sort=trending)
264279
- [LLaMA](https://huggingface.co/models?sort=trending&search=llama+gguf)
265280

266-
You can either manually download the GGUF file or directly use any `llama.cpp`-compatible models from [Hugging Face](https://huggingface.co/) or other model hosting sites, such as [ModelScope](https://modelscope.cn/), by using this CLI argument: `-hf <user>/<model>[:quant]`.
281+
You can either manually download the GGUF file or directly use any `llama.cpp`-compatible models from [Hugging Face](https://huggingface.co/) or other model hosting sites, such as [ModelScope](https://modelscope.cn/), by using this CLI argument: `-hf <user>/<model>[:quant]`. For example:
282+
283+
```sh
284+
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
285+
```
267286

268287
By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable `MODEL_ENDPOINT`. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. `MODEL_ENDPOINT=https://www.modelscope.cn/`.
269288

docs/build.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Build llama.cpp locally
22

3+
The main product of this project is the `llama` library. Its C-style interface can be found in [include/llama.h](include/llama.h).
4+
5+
The project also includes many example programs and tools using the `llama` library. The examples range from simple, minimal code snippets to sophisticated sub-projects such as an OpenAI-compatible HTTP server.
6+
37
**To get the Code:**
48

59
```bash

docs/install.md

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,42 @@
11
# Install pre-built version of llama.cpp
22

3-
## Homebrew
3+
| Install via | Windows | Mac | Linux |
4+
|-------------|---------|-----|-------|
5+
| Winget || | |
6+
| Homebrew | |||
7+
| MacPorts | || |
8+
| Nix | |||
49

5-
On Mac and Linux, the homebrew package manager can be used via
10+
## Winget (Windows)
11+
12+
```sh
13+
winget install llama.cpp
14+
```
15+
16+
The package is automatically updated with new `llama.cpp` releases. More info: https://github.com/ggml-org/llama.cpp/issues/8188
17+
18+
## Homebrew (Mac and Linux)
619

720
```sh
821
brew install llama.cpp
922
```
23+
1024
The formula is automatically updated with new `llama.cpp` releases. More info: https://github.com/ggml-org/llama.cpp/discussions/7668
1125

12-
## MacPorts
26+
## MacPorts (Mac)
1327

1428
```sh
1529
sudo port install llama.cpp
1630
```
17-
see also: https://ports.macports.org/port/llama.cpp/details/
1831

19-
## Nix
32+
See also: https://ports.macports.org/port/llama.cpp/details/
2033

21-
On Mac and Linux, the Nix package manager can be used via
34+
## Nix (Mac and Linux)
2235

2336
```sh
2437
nix profile install nixpkgs#llama-cpp
2538
```
39+
2640
For flake enabled installs.
2741

2842
Or
@@ -34,13 +48,3 @@ nix-env --file '<nixpkgs>' --install --attr llama-cpp
3448
For non-flake enabled installs.
3549

3650
This expression is automatically updated within the [nixpkgs repo](https://github.com/NixOS/nixpkgs/blob/nixos-24.05/pkgs/by-name/ll/llama-cpp/package.nix#L164).
37-
38-
## Flox
39-
40-
On Mac and Linux, Flox can be used to install llama.cpp within a Flox environment via
41-
42-
```sh
43-
flox install llama-cpp
44-
```
45-
46-
Flox follows the nixpkgs build of llama.cpp.

0 commit comments

Comments
 (0)