What LocalAI is, and what it needs from a server
LocalAI is one server program that answers the OpenAI API on your own hardware, with a web interface on the same port. Chat, embeddings, images and speech each run in a backend (llama.cpp, whisper.cpp, diffusers, vLLM and others) downloaded only when a model you install needs it.
- Docker or Podman. The project points servers and every GPU image to containers.
- The vendor's driver on the host. For NVIDIA, the driver plus the NVIDIA Container Toolkit; for AMD, a ROCm-supported kernel driver.
- Disk. The LocalAI FAQ suggests two to three times a model's size free, on SSD.
- Card memory. It decides which models you can serve at all.
Which image matches your card?
Pick the tag for the accelerator in the machine; the CPU-only image would leave the card idle. AMD owners can read about Instinct cards on bare metal.
| Your card | Image tag | Passed to Docker |
|---|---|---|
| NVIDIA, driver supports CUDA 13 | localai/localai:latest-gpu-nvidia-cuda-13 | --gpus all |
| NVIDIA, CUDA 12 driver, or a Maxwell, Pascal or Volta card | localai/localai:latest-gpu-nvidia-cuda-12 | --gpus all |
| AMD Instinct or Radeon on a target the LocalAI GPU page lists (MI210 yes; MI50, gfx906, no) | localai/localai:latest-gpu-hipblas | --device=/dev/kfd --device=/dev/dri --group-add=video |
| Intel GPU | localai/localai:latest-gpu-intel | --device /dev/dri |
| Anything else Vulkan can drive | localai/localai:latest-gpu-vulkan | the card's own flags: --gpus all (NVIDIA) or --device /dev/dri (AMD, Intel) |
NVIDIA removed library support for Maxwell, Pascal and Volta cards in CUDA 13. If your card is one of those (a Tesla M60, P4, P40 or V100, a GTX 1080, a TITAN V), stay on the CUDA 12 image. Kepler and older cards (such as a Quadro K3100M or 3000M) are outside CUDA 12 altogether, so neither CUDA image uses them. nvidia-smi prints the highest CUDA version your driver supports in its top line. The prebuilt ROCm image does not cover the Instinct MI50 (gfx906): AMD removed rocBLAS support for it in ROCm 6.4.
How much card memory do your models need?
Start from the weights: parameters times bytes per parameter, which is half a byte each at 4-bit. That is the floor, not the requirement, because the runtime and the key-value cache (the memory that holds the conversation so far) share the card. Budget about 1.25 times the weights for one user at a time, and about 1.5 times for serving many requests at once.
| Model size | Weights at 4-bit | One user at a time | Card memory that clears it |
|---|---|---|---|
| 7-8 billion | about 5 GB | about 6.5 GB | 8 GB |
| 13-14 billion | about 8 GB | about 10 GB | 11-12 GB, or 16 GB for headroom |
| 30-34 billion | about 19 GB | about 24 GB | 24 GB is tight; 32 GB or more is comfortable |
| 70 billion | about 38 GB | about 48 GB | 48 GB is tight; more for serving traffic |
LocalAI adds two things to that arithmetic. A model stays loaded once used, so chat, embedding and speech models pile up on the card unless you cap them: LOCALAI_MAX_ACTIVE_BACKENDS=1 unloads the least recently used one first. And context_size in a model's YAML grows the key-value cache, so a long context can turn a fitting model into an out-of-memory error. The project's fixes: a smaller quantisation, a shorter context_size, fewer layers on the card (gpu_layers), or freeing memory other processes hold. Our guide to sizing a language model for a card works model by model.
Why a whole server suits it
The image choice above depends on a driver you control, and on our GPU dedicated servers you do: the card sits in a physical machine rented to one account and is handed straight to your operating system. You pin your own driver and CUDA or ROCm release, with no hypervisor, MIG partition or time-slicing between you and the card.
- Model downloads cost nothing extra. The port is unmetered both ways, so pulling tens of gigabytes of weights never shows up as traffic on the invoice.
- A driver mistake is recoverable. The server's own HPE iLO or Dell iDRAC console shows you the screen if a driver update takes the network down.
- Three shelves of cards. Data-centre accelerators with ECC memory, professional cards, and consumer GeForce cards, each priced as a line beside the machine.
- No contract. One month is the default term. A machine already carrying your card goes live in about 30 minutes once approved; fitting a card we hold takes 4 to 24 hours, and one bought in for you 5 to 10 working days or longer.
Up and running in an evening
These steps assume an NVIDIA card on Ubuntu or Debian, with the driver, Docker and the NVIDIA Container Toolkit installed from their vendors' guides (the toolkit's ends with sudo nvidia-ctk runtime configure --runtime=docker and a Docker restart).
- Check that containers see the card, then create a folder and a random API key only your user can read, kept in a file rather than on a command line.
- Save the Compose file below as
compose.yamlthere. It is adapted from the project's Compose examples: a CUDA 12 image, a small gallery model, a restart policy, the port bound to 127.0.0.1, the key from.env, all four data paths on volumes and one model loaded at a time, with the legacy NVIDIA device block. - Start it, watch the first model download, and confirm the API refuses a request without the key and answers one with it.
- From your own computer, open an SSH tunnel,
ssh -N -L 8080:127.0.0.1:8080 you@your-server, then browse to http://localhost:8080. Clients usehttp://localhost:8080/v1with the key as bearer token.
# 1. Confirm Docker can reach the card (check from the LocalAI docs)
docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
# 2. A folder for the stack, and an API key file only you can read
mkdir -p ~/localai && cd ~/localai && ( umask 077; printf 'LOCALAI_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env )
services:
api:
image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
restart: unless-stopped
ports:
- "127.0.0.1:8080:8080" # loopback only: nothing public on 8080
env_file:
- .env # holds LOCALAI_API_KEY
environment:
- LOCALAI_MAX_ACTIVE_BACKENDS=1
volumes:
- models:/models
- backends:/backends
- configuration:/configuration
- data:/data
command:
- llama-3.2-1b-instruct:q4_k_m
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu, compute, utility]
volumes:
models:
backends:
configuration:
data:
docker compose up -d
docker compose logs -f api # Ctrl+C once the model has finished downloading
# Without the key the API should refuse you; with it, it lists your models
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
set -a; . ./.env; set +a
curl -s http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LOCALAI_API_KEY"
Never publish 8080 on the public address without a key: with no key and no user accounts, LocalAI restricts nothing, and Docker's published ports bypass ufw. Keep it on loopback, or put an HTTPS reverse proxy in front if other machines must call it. If you add a firewall, allow SSH before enabling it.
If the container stops with an nvidia-container-cli invalid-expression error, the LocalAI docs suggest the CDI form, driver: nvidia.com/gpu, which needs NVIDIA Container Toolkit 1.14 or later.
Keeping it running well
- Updates.
docker compose pullthendocker compose up -d. Files outside the four volumes are lost when the container is recreated. Where others depend on it, pin a released tag such asv5.0.0-gpu-nvidia-cuda-12instead oflatest. - Rotating the key. Edit
.env, thendocker compose up -d --force-recreate; a plain restart keeps the old value. Comma-separated keys are accepted, so add the new one, move clients, then drop the old. - Who holds the key. A key grants full admin rights, model installs and deletes included. For separate users and roles there are user accounts (
LOCALAI_AUTH=true); the first account registered becomes admin, so register yours while the port is still on loopback. - Backups. Models can be downloaded again; the
configurationvolume (keys, runtime settings) anddatavolume (chats, collections) cannot. Stop the stack and archive those two. - Watching it.
/readyzand/healthzanswer without a key, so a monitor can poll them. On a failure the server log has the backend's real reason;DEBUG=trueadds more.
When this is the wrong place to run it
A model too big for one card is the first limit. The largest card we fit holds 96 GB; past that it is several cards or machines, quoted rather than configured, and multi-GPU and cluster builds covers it. Not every accelerator we fit has a LocalAI image, either: check the table above before choosing an unusual card.
Prebuilt backends can also stop with an illegal-instruction (SIGILL) error on a processor lacking an instruction set they were built for, such as AVX2. Some cards sit in older Xeon E5 hosts; grep -c avx2 /proc/cpuinfo prints 0 if yours lacks it, and several cards fit more than one chassis. And if you send a few requests a week, a plain dedicated server running small models on the CPU image may be all the job needs.
Questions people ask
- Does LocalAI need a graphics card at all?
- No. The plain
localai/localai:latestimage runs on the processor alone, which suits small models. A card makes larger models and several users practical, because weights and key-value cache sit in card memory instead of system RAM. - Will apps written for OpenAI work against it unchanged?
- Usually, yes. Many OpenAI clients read the endpoint from
OPENAI_BASE_URL(older ones fromOPENAI_API_BASE), so pointing it at your server's/v1address and passing your LocalAI key as the API key is often all the change they need. Use the model name that/v1/modelsreturns. - Can I keep a chat model and an embedding model loaded together?
- Yes, if both fit in card memory at once. Raise
LOCALAI_MAX_ACTIVE_BACKENDSto 2, or use per-model concurrency groups so only the heavy models evict each other. If loads start failing with out-of-memory errors, lower it again or shortencontext_size. - Can I use an AMD Instinct card instead of NVIDIA?
- Yes for the MI210 (gfx90a) and the other targets the LocalAI GPU page lists, with the
latest-gpu-hipblasimage and/dev/kfdand/dev/dripassed into the container. The prebuilt image does not cover the MI50 (gfx906). Check the card and the host'samdgpudriver against the ROCm compatibility matrix the LocalAI GPU page points to.