Workload · Running LocalAI on your own card

LocalAI Self Hosted: An OpenAI-Compatible API on Your Own GPU

To run LocalAI self hosted, start the container image built for your accelerator (CUDA 12, CUDA 13, ROCm, Intel or Vulkan), give it four persistent volumes and an API key, and point any OpenAI client at port 8080. After that, card memory makes most of the decisions: which models load, how many stay loaded together, and how long a context you can afford.

The short version
  1. One image per accelerator; the tag must match your driver
  2. Set LOCALAI_API_KEY before the first start, or the API is open
  3. Card memory decides model size, context and how many stay loaded
  4. Whole server, card passed straight to your own OS
  5. Unmetered port for pulling model weights, monthly term

What LocalAI is, and what it needs from a server

LocalAI is one server program that answers the OpenAI API on your own hardware, with a web interface on the same port. Chat, embeddings, images and speech each run in a backend (llama.cpp, whisper.cpp, diffusers, vLLM and others) downloaded only when a model you install needs it.

  • Docker or Podman. The project points servers and every GPU image to containers.
  • The vendor's driver on the host. For NVIDIA, the driver plus the NVIDIA Container Toolkit; for AMD, a ROCm-supported kernel driver.
  • Disk. The LocalAI FAQ suggests two to three times a model's size free, on SSD.
  • Card memory. It decides which models you can serve at all.

Which image matches your card?

Pick the tag for the accelerator in the machine; the CPU-only image would leave the card idle. AMD owners can read about Instinct cards on bare metal.

Your cardImage tagPassed to Docker
NVIDIA, driver supports CUDA 13localai/localai:latest-gpu-nvidia-cuda-13--gpus all
NVIDIA, CUDA 12 driver, or a Maxwell, Pascal or Volta cardlocalai/localai:latest-gpu-nvidia-cuda-12--gpus all
AMD Instinct or Radeon on a target the LocalAI GPU page lists (MI210 yes; MI50, gfx906, no)localai/localai:latest-gpu-hipblas--device=/dev/kfd --device=/dev/dri --group-add=video
Intel GPUlocalai/localai:latest-gpu-intel--device /dev/dri
Anything else Vulkan can drivelocalai/localai:latest-gpu-vulkanthe card's own flags: --gpus all (NVIDIA) or --device /dev/dri (AMD, Intel)

NVIDIA removed library support for Maxwell, Pascal and Volta cards in CUDA 13. If your card is one of those (a Tesla M60, P4, P40 or V100, a GTX 1080, a TITAN V), stay on the CUDA 12 image. Kepler and older cards (such as a Quadro K3100M or 3000M) are outside CUDA 12 altogether, so neither CUDA image uses them. nvidia-smi prints the highest CUDA version your driver supports in its top line. The prebuilt ROCm image does not cover the Instinct MI50 (gfx906): AMD removed rocBLAS support for it in ROCm 6.4.

How much card memory do your models need?

Start from the weights: parameters times bytes per parameter, which is half a byte each at 4-bit. That is the floor, not the requirement, because the runtime and the key-value cache (the memory that holds the conversation so far) share the card. Budget about 1.25 times the weights for one user at a time, and about 1.5 times for serving many requests at once.

Model sizeWeights at 4-bitOne user at a timeCard memory that clears it
7-8 billionabout 5 GBabout 6.5 GB8 GB
13-14 billionabout 8 GBabout 10 GB11-12 GB, or 16 GB for headroom
30-34 billionabout 19 GBabout 24 GB24 GB is tight; 32 GB or more is comfortable
70 billionabout 38 GBabout 48 GB48 GB is tight; more for serving traffic

LocalAI adds two things to that arithmetic. A model stays loaded once used, so chat, embedding and speech models pile up on the card unless you cap them: LOCALAI_MAX_ACTIVE_BACKENDS=1 unloads the least recently used one first. And context_size in a model's YAML grows the key-value cache, so a long context can turn a fitting model into an out-of-memory error. The project's fixes: a smaller quantisation, a shorter context_size, fewer layers on the card (gpu_layers), or freeing memory other processes hold. Our guide to sizing a language model for a card works model by model.

Why a whole server suits it

The image choice above depends on a driver you control, and on our GPU dedicated servers you do: the card sits in a physical machine rented to one account and is handed straight to your operating system. You pin your own driver and CUDA or ROCm release, with no hypervisor, MIG partition or time-slicing between you and the card.

  • Model downloads cost nothing extra. The port is unmetered both ways, so pulling tens of gigabytes of weights never shows up as traffic on the invoice.
  • A driver mistake is recoverable. The server's own HPE iLO or Dell iDRAC console shows you the screen if a driver update takes the network down.
  • Three shelves of cards. Data-centre accelerators with ECC memory, professional cards, and consumer GeForce cards, each priced as a line beside the machine.
  • No contract. One month is the default term. A machine already carrying your card goes live in about 30 minutes once approved; fitting a card we hold takes 4 to 24 hours, and one bought in for you 5 to 10 working days or longer.

Up and running in an evening

These steps assume an NVIDIA card on Ubuntu or Debian, with the driver, Docker and the NVIDIA Container Toolkit installed from their vendors' guides (the toolkit's ends with sudo nvidia-ctk runtime configure --runtime=docker and a Docker restart).

  1. Check that containers see the card, then create a folder and a random API key only your user can read, kept in a file rather than on a command line.
  2. Save the Compose file below as compose.yaml there. It is adapted from the project's Compose examples: a CUDA 12 image, a small gallery model, a restart policy, the port bound to 127.0.0.1, the key from .env, all four data paths on volumes and one model loaded at a time, with the legacy NVIDIA device block.
  3. Start it, watch the first model download, and confirm the API refuses a request without the key and answers one with it.
  4. From your own computer, open an SSH tunnel, ssh -N -L 8080:127.0.0.1:8080 you@your-server, then browse to http://localhost:8080. Clients use http://localhost:8080/v1 with the key as bearer token.
# 1. Confirm Docker can reach the card (check from the LocalAI docs)
docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

# 2. A folder for the stack, and an API key file only you can read
mkdir -p ~/localai && cd ~/localai && ( umask 077; printf 'LOCALAI_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env )
services:
  api:
    image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"   # loopback only: nothing public on 8080
    env_file:
      - .env                    # holds LOCALAI_API_KEY
    environment:
      - LOCALAI_MAX_ACTIVE_BACKENDS=1
    volumes:
      - models:/models
      - backends:/backends
      - configuration:/configuration
      - data:/data
    command:
      - llama-3.2-1b-instruct:q4_k_m
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu, compute, utility]
volumes:
  models:
  backends:
  configuration:
  data:
docker compose up -d
docker compose logs -f api     # Ctrl+C once the model has finished downloading

# Without the key the API should refuse you; with it, it lists your models
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
set -a; . ./.env; set +a
curl -s http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LOCALAI_API_KEY"

Never publish 8080 on the public address without a key: with no key and no user accounts, LocalAI restricts nothing, and Docker's published ports bypass ufw. Keep it on loopback, or put an HTTPS reverse proxy in front if other machines must call it. If you add a firewall, allow SSH before enabling it.

If the container stops with an nvidia-container-cli invalid-expression error, the LocalAI docs suggest the CDI form, driver: nvidia.com/gpu, which needs NVIDIA Container Toolkit 1.14 or later.

Keeping it running well

  • Updates. docker compose pull then docker compose up -d. Files outside the four volumes are lost when the container is recreated. Where others depend on it, pin a released tag such as v5.0.0-gpu-nvidia-cuda-12 instead of latest.
  • Rotating the key. Edit .env, then docker compose up -d --force-recreate; a plain restart keeps the old value. Comma-separated keys are accepted, so add the new one, move clients, then drop the old.
  • Who holds the key. A key grants full admin rights, model installs and deletes included. For separate users and roles there are user accounts (LOCALAI_AUTH=true); the first account registered becomes admin, so register yours while the port is still on loopback.
  • Backups. Models can be downloaded again; the configuration volume (keys, runtime settings) and data volume (chats, collections) cannot. Stop the stack and archive those two.
  • Watching it. /readyz and /healthz answer without a key, so a monitor can poll them. On a failure the server log has the backend's real reason; DEBUG=true adds more.

When this is the wrong place to run it

A model too big for one card is the first limit. The largest card we fit holds 96 GB; past that it is several cards or machines, quoted rather than configured, and multi-GPU and cluster builds covers it. Not every accelerator we fit has a LocalAI image, either: check the table above before choosing an unusual card.

Prebuilt backends can also stop with an illegal-instruction (SIGILL) error on a processor lacking an instruction set they were built for, such as AVX2. Some cards sit in older Xeon E5 hosts; grep -c avx2 /proc/cpuinfo prints 0 if yours lacks it, and several cards fit more than one chassis. And if you send a few requests a week, a plain dedicated server running small models on the CPU image may be all the job needs.

Questions people ask

Does LocalAI need a graphics card at all?
No. The plain localai/localai:latest image runs on the processor alone, which suits small models. A card makes larger models and several users practical, because weights and key-value cache sit in card memory instead of system RAM.
Will apps written for OpenAI work against it unchanged?
Usually, yes. Many OpenAI clients read the endpoint from OPENAI_BASE_URL (older ones from OPENAI_API_BASE), so pointing it at your server's /v1 address and passing your LocalAI key as the API key is often all the change they need. Use the model name that /v1/models returns.
Can I keep a chat model and an embedding model loaded together?
Yes, if both fit in card memory at once. Raise LOCALAI_MAX_ACTIVE_BACKENDS to 2, or use per-model concurrency groups so only the heavy models evict each other. If loads start failing with out-of-memory errors, lower it again or shorten context_size.
Can I use an AMD Instinct card instead of NVIDIA?
Yes for the MI210 (gfx90a) and the other targets the LocalAI GPU page lists, with the latest-gpu-hipblas image and /dev/kfd and /dev/dri passed into the container. The prebuilt image does not cover the MI50 (gfx906). Check the card and the host's amdgpu driver against the ROCm compatibility matrix the LocalAI GPU page points to.

Put LocalAI on a card of your own

Choose the card your models need, see the whole machine priced around it, and run your own OpenAI-compatible API on a port nobody meters.

Get a GPU server for LocalAI

Read next