A llama.cpp server for your own private model API

What you end up with

One program, llama-server, holding one model in memory and listening on a port. Your chatbot, your internal tool or your coding assistant sends it requests at /v1/chat/completions and gets answers back, with no per-token bill and no prompts leaving your machine. It also ships a small web chat page at the same address, handy for checking that the model behaves before you wire up an app.

The model comes as a GGUF file: the weights already squeezed to fewer bits per number (quantised), so they take less memory. You download one from Hugging Face, or make your own with the llama-quantize tool that comes with the project.

Will your model fit on the card?

Yes, if the file plus some working room is smaller than the card's memory. The project's own measurements give the file sizes; the right-hand column is our reading of them.

The working room is the conversation memory (the KV cache), and it grows with how long a conversation can get and how many people talk to the model at once. Our guide to running a model budgets about a quarter on top of the file for one user at a time with llama.cpp. If you plan for several users, leave more.

Q4_K_M is the usual starting point: llama-server picks it by default when you name a Hugging Face repo without a quantisation. Go up to Q5 or Q6 if the card has room and you notice the answers getting sloppy.

Getting the card to do the work

The flag that matters is -ngl, short for GPU layers. A model is a stack of layers; every layer you place on the card runs fast, every layer left behind runs on the processor from ordinary RAM. Ask for all of them (-ngl 999) and llama.cpp puts as many on the card as there are.

If the model is bigger than the card, lower the number. It still runs, slower, with the rest of the model in system memory. This split between card and RAM is what lets a modest card carry a model it could not hold alone. Recent builds also switch on --fit by default, which trims settings you left unset, such as context length, until the model fits.

Nothing happening on the card? Run llama-server --list-devices. If your GPU is missing from that list, the program was built without CUDA support, and no flag will fix it: rebuild it, or use the CUDA container below.

Two cards in one machine are used together automatically: each takes a share of the layers, in proportion to its memory unless you set -ts, for example -ts 1,1 for an even split. A machine with more than one card is something we quote for you; our page on multi-card servers explains the options.

Up and running in an evening

1. Install the NVIDIA driver on your server. The container route below also needs Docker and NVIDIA's Container Toolkit, so Docker can hand the card to the container.
2. Make a folder for models and put your GGUF file in it.
3. Choose a long random key and save it in a file only root can read.
4. Start the official CUDA server image, published on localhost only, with the key file and your settings.
5. Ask /health whether the model has loaded, then send a first chat request with the key.

docker run -d --name llama --restart unless-stopped --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v /srv/models:/models -v /etc/llama:/etc/llama:ro \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/my-model-Q4_K_M.gguf -ngl 999 -c 16384 -np 2 \
  --host 0.0.0.0 --port 8080 --api-key-file /etc/llama/keys

curl -s localhost:8080/health
curl -s localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $(sudo grep -v '^#' /etc/llama/keys | head -1)" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Are you there?"}]}'

Prefer no Docker? Build it yourself: clone the repository, run cmake -B build -DGGML_CUDA=ON, then cmake --build build --config Release, and the program lands in build/bin.

Serving several people at once

-np is the number of slots, meaning requests the server works on together; -c is the size of the conversation memory they all share. Four slots on a small -c gives each person a short memory, so raise both together and watch the card's memory as you go.

• Leaving -c unset uses whatever the model claims as its maximum, which can be huge. Set it to what your app really needs.
• Storing the cache at 8 bits (--cache-type-k q8_0 plus --cache-type-v q8_0) roughly halves its size, at a small cost in accuracy.
• Need more than one model? Start the server without -m and point --models-dir at a folder of GGUF files; it loads whichever model a request names.

Keeping strangers out

Out of the box llama-server only listens on the machine itself, and that is the safe default. Without --api-key or --api-key-file any visitor who finds the address can use your model as much as they like. Before opening it to the internet, set a key and put it behind a reverse proxy, as the project recommends, and have that proxy serve HTTPS.

Keep the newer agent features (--tools, --agent, MCP servers) switched off on anything public: they let the model read files and run commands on your server. And always pass --port yourself, because the default has changed between versions.

The server that fits

A llama.cpp server goes wrong in two ways: the card runs out of memory, or something else on the machine slows it down. Our GPU servers rule out the second. The card sits in a physical machine rented to you alone and is handed directly to your own operating system: no slice, no time-sharing, no other tenant whose job makes your replies slow at busy hours. You install the driver and CUDA or ROCm version you want and keep it there.

• Filter the list by the memory your GGUF needs; each row is a complete machine with that card in it.
• Every core, all the RAM and root are yours, plus a remote console (iLO or iDRAC) if a driver update takes the network away.
• The port is unmetered, so downloading a 40 GB model or answering requests all day adds nothing.
• Monthly, no contract. A machine that already carries your card is ready about 30 minutes after approval; fitting a card we hold takes 4 to 24 hours.
• Five cities in the US and Europe: put the server near your users, using our data center list.

Is llama.cpp the right engine for you?

Choose it when you want GGUF files, control over every setting, a model slightly too big for your card, or an older card. If you would rather fetch models by name and skip the flags, Ollama on a server of your own is simpler. If dozens of people will use the model at the same moment and the card holds it comfortably, read our vLLM workload page before deciding.

Questions people ask

Why is my llama.cpp server running on the processor instead of the card?

Usually because the program was built without GPU support or the driver is not loaded. llama-server --list-devices shows which cards it can use. If yours is listed, check that -ngl was not set to a low number.

Can I use my existing OpenAI code with it?

Yes. Change the base URL to your server's address followed by /v1 and use your llama-server key as the API key. Chat completions, completions and the models list answer on the familiar paths; for embeddings, run a second llama-server with an embedding model and --embeddings.

What size card do I need for a 70B model?

Its common 4-bit file (Q4_K_M) is about 43 GB, so a 48 GB card holds it with a short context and few users; for long conversations or several users, add a second card or put some layers in system RAM, which is slower.

Can I run it without a graphics card?

Yes, llama.cpp runs on the processor alone. It suits a small 4-bit model with one or two users at a time; once people are waiting on replies, moving the layers onto a card is what makes them fast.

Does the server restart by itself after a reboot?

Only if you set it up that way. With Docker, --restart unless-stopped brings the container back after a reboot; if you built from source, run it as a systemd service.