AnythingLLM self hosted for your team, on a GPU server you control

What your team gets

AnythingLLM turns a pile of documents into something you can talk to. You create a workspace, drop PDFs, notes or web pages into it, and the app splits them up, stores them in its own search index (a vector database called LanceDB, already inside the app) and quotes from them when someone asks a question.

There are two editions. The Desktop app is meant for one person on one computer. For a server you want the Docker edition: several people at once, each with their own login, admin control over who sees which workspace, chat widgets you can embed on a website, and your own branding. Both are open source under the MIT licence. This is how we read it, and it is no substitute for legal advice. You can use and change it freely, and any copies you hand to others must carry the original copyright line and the MIT permission text.

Will your server run it?

Almost certainly, if the processor is new enough. The project asks for very little: about 2 GB of memory, a two-core processor and 5 GB of disk as a starting point, plus room for your documents and any models you download.

The catch is a processor feature called AVX2. LanceDB needs it, and on an Intel or AMD chip without it the app crashes the moment it starts, reporting an Illegal instruction. Among Intel's server chips, AVX2 arrived with the Haswell generation (from 2013); not every Intel chip since has it. That matters on our GPU list: its cheapest machine half is a Xeon E5-2600 v1/v2, which is older than that. Choose a card on a Xeon E5-2620 v4, a Xeon Silver or an AMD EPYC machine instead. Several cards fit more than one machine, and the configurator shows which ones each card comes on. Take yours on a machine with AVX2, and if a card is only offered on the E5-2600 v1/v2, go for another card.

• Card: for the built-in model engine on NVIDIA, the Docker guide asks for a Turing-generation (RTX 20 series) card or newer. Plenty of older cards on the list do not qualify.
• Card memory: the model has to fit in it. Our LLM hosting page shows how much a 7B to 70B model needs.
• Disk: each model you download through AnythingLLM takes a few more gigabytes in the same folder.
• Conversation length: if you leave the context window empty, the built-in engine uses the model's own limit but never more than 16,384 tokens, to avoid running out of memory.

Up and running: the setup

You need Ubuntu 24.04 with three things installed first: the graphics driver from NVIDIA, Docker Engine installed from Docker's own apt repo, plus NVIDIA's Container Toolkit (hook it into Docker by running its nvidia-ctk tool's runtime configure command for Docker, then restart Docker). After that, paste this as root. It turns the firewall on, checks the processor, the Docker version and the GPU hookup, and stops with a message if any of them is wrong:

sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw && ufw allow OpenSSH && ufw --force enable
A=/var/lib/anythingllm
ver=$(docker info 2>/dev/null | sed -n 's/^ *Server Version: *\([0-9]*\).*/\1/p')
if ! grep -qw avx2 /proc/cpuinfo; then echo "STOP: no AVX2 on this processor"
elif ! [ "${ver:-0}" -ge 28 ] 2>/dev/null; then echo "STOP: Docker Engine is older than 28"
elif ! docker info 2>/dev/null | grep -qi 'runtimes:.*nvidia'; then echo "STOP: Docker cannot see the NVIDIA runtime"
elif [ -e $A/.env ]; then echo "STOP: $A was set up before, leaving it alone"
else
mkdir -p $A && install -m 600 /dev/null $A/.env &&
echo 'DISABLE_TELEMETRY="true"' > $A/.env && chown -R 1000:1000 $A &&
cat > /root/anythingllm-compose.yaml <<'EOF' && docker compose -f /root/anythingllm-compose.yaml up -d
services:
  anythingllm:
    image: mintplexlabs/anythingllm:1.19.0
    container_name: anythingllm
    restart: unless-stopped
    ports: ["127.0.0.1:3001:3001"]
    cap_add: [SYS_ADMIN]
    environment: [STORAGE_DIR=/app/server/storage, ENABLE_BUILT_IN_LLM=true]
    volumes:
      - /var/lib/anythingllm:/app/server/storage
      - /var/lib/anythingllm/.env:/app/server/.env
    extra_hosts: ["host.docker.internal:host-gateway"]
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]
EOF
fi

What we changed from the project's own command, so nothing surprises you: the app listens on 127.0.0.1 alone (the server itself), the image is pinned to one release rather than latest, Docker restarts it after a reboot instead of removing it (--rm), it has a fixed name, the .env settings file starts private (mode 600) with telemetry switched off, and host.docker.internal is mapped the way the docs suggest on Linux. The GPU part comes from the commented NVIDIA option in the project's HOW_TO_USE_DOCKER.md. SYS_ADMIN stays: the cloud setup guide says the website scraper runs a sandboxed Chromium that requires it.

Look after that .env. AnythingLLM writes its own encryption key into it on first start, and a login-signing secret once multi-user mode is on; the source files are server/utils/EncryptionManager/index.js and server/endpoints/system.js. If it is lost, saved secrets can no longer be read.

Two details explain the odd-looking steps. The folder is handed to user 1000 because the app inside the container is always user 1000; the documentation says to use chown, never chmod 777, which would let every account on the server read your documents. And the 127.0.0.1 binding matters because a Docker port published without an address is open on every address, 0.0.0.0 and [::] alike, and its IPv4 rules let that traffic slip past ufw (IPv6 traffic too, if you have switched IPv6 on inside Docker itself). Docker keeps a 127.0.0.1 port truly private only from Engine 28 onward, so the block refuses anything older.

Make yourself the admin before anyone else can

AnythingLLM starts out in single-user mode. In that mode a password is optional, so anyone who opens the page could change its settings and read every chat. Keep it private and claim it first.

1. Give it a minute, until docker ps lists the container as (healthy).
2. On your own computer, open a tunnel: ssh -L 8001:127.0.0.1:3001 you@your-server. Leave it open.
3. Visit http://localhost:8001 in your browser. During onboarding, choose Built-in as the model provider and download a model that fits your card. Your first chat is slower because it also fetches the right llama.cpp runtime.
4. In the security settings, switch on Enable multi-user mode, then create the admin username and password, then sign in again. You cannot turn multi-user mode back off, so this is a one-way door.
5. Invite your colleagues. A Manager can view and manage every workspace, but not the LLM, embedder or vector database settings; a Default user only chats in the workspaces you add them to.

Let your team in over HTTPS

Now give it a proper address with a padlock. Make a domain name resolve to this server (its IPv4 address record, plus the IPv6 one if your server has IPv6), then let Caddy fetch the certificate and pass traffic to the app. Replace docs.example.com with your own name before pasting:

apt-get update && apt-get install -y caddy &&
cat > /etc/caddy/Caddyfile <<'EOF' &&
docs.example.com {
	reverse_proxy 127.0.0.1:3001 {
		flush_interval -1
	}
}
EOF
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile &&
ufw allow 80,443/tcp && systemctl reload caddy

AnythingLLM's own example uses nginx; this is our Caddy version of it. Their nginx file carries WebSocket upgrades for the agent feature and switches off response buffering so answers stream word by word. Caddy handles WebSockets on its own, and flush_interval -1 does the buffering part. Caddy is part of Ubuntu 24.04. Caddy is a normal program on the server rather than a container, so ufw only lets visitors reach it once ports 80 and 443 are allowed; port 3001 stays shut.

Already running Ollama on the same machine?

Then you can skip the built-in engine and pick Ollama as the provider. One trap catches nearly everyone: Ollama listening on 127.0.0.1 cannot be reached from inside the container. The Docker guide's answer on Linux is the Docker bridge address, http://172.17.0.1:11434. Start Ollama with OLLAMA_HOST=172.17.0.1, let only Docker's private range reach it with ufw allow proto tcp from 172.16.0.0/12 to 172.17.0.1 port 11434, and enter http://172.17.0.1:11434 as the Ollama address in AnythingLLM. More on that setup is on our Ollama server page.

Keeping your documents safe

Everything lives in /var/lib/anythingllm: the SQLite database with users and chats, your uploaded files, the search index, downloaded models and the .env. Pause the app so the files are not changing, pack up that folder plus the compose file, then start it again:

A=/var/lib/anythingllm; T=/root/backup/allm-$(date +%F-%H%M%S).tar.gz
mkdir -p -m 700 /root/backup &&
docker compose -f /root/anythingllm-compose.yaml stop &&
(umask 077; tar -czf "$T" -C / var/lib/anythingllm root/anythingllm-compose.yaml); rc=$?
docker compose -f /root/anythingllm-compose.yaml start
[ "$rc" -eq 0 ] && tar -tzf "$T" > /dev/null && echo "ok: $T"

Keep only archives that end with ok, and copy them to a different machine (scp, or rsync -a without --delete). They contain your encryption key, so store them somewhere private. AnythingLLM and its data are yours to run and to back up. Phone and chat support is included around the clock on every machine, and server management has a free Basic tier with monitoring, OS reloads and part swaps if you want more help.

To restore, as root:

1. Check the archive opens: tar -tzf /root/backup/allm-STAMP.tar.gz > /dev/null && echo ok, with STAMP being the date and time in its name. Without ok, this archive is no good; choose another. Otherwise take the app down: docker compose -f /root/anythingllm-compose.yaml down.
2. If /var/lib/anythingllm.old already exists, sort that out before going any further. Otherwise move today's data out of the way with mv /var/lib/anythingllm /var/lib/anythingllm.old, and keep a copy of today's compose file with cp /root/anythingllm-compose.yaml /root/anythingllm-compose.yaml.old; together they are your way back.
3. Unpack the data and the compose file it was taken with: tar -xzf /root/backup/allm-STAMP.tar.gz -C / var/lib/anythingllm root/anythingllm-compose.yaml. Make sure /var/lib/anythingllm/anythingllm.db is there, and run chown -R 1000:1000 /var/lib/anythingllm.
4. Bring it up again (docker compose -f /root/anythingllm-compose.yaml up -d), and once it reports healthy, sign in and look through your workspaces.
5. Happy? Delete /var/lib/anythingllm.old. Not happy? Take it down, rename the restored copy to something like anythingllm.bad, mv /var/lib/anythingllm.old /var/lib/anythingllm, mv /root/anythingllm-compose.yaml.old /root/anythingllm-compose.yaml, and bring it up.

For a new version, back up, edit the release number on the image: line, then run docker compose -f /root/anythingllm-compose.yaml pull followed by up -d. Each start applies any database migrations (the npx prisma migrate deploy step in docker-entrypoint.sh). So if an update goes wrong, restore the backup you took just before it: the steps above bring back the old data and the compose file with the old release on its image: line, together. Never switch the tag back on its own, because an older release may not understand the upgraded database. The project does not say this; it is our own caution.

Why a whole GPU server suits it

Your documents and your model sit on one physical machine that nobody else uses. On a Primcast GPU server the card goes into a PCIe slot and is handed to your operating system directly, with your own driver and CUDA release; no hypervisor, no MIG partition and no time-slicing means nobody else's job slows your answers. Nothing counts the bytes on the port, so multi-gigabyte model downloads and big document uploads never show up on the invoice. And the server's own HPE iLO or Dell iDRAC gives you its screen, virtual media and the power button even when a bad graphics update knocks the server off the network.

• When it arrives, counted from when the order is approved: about 30 minutes if a machine already carries the card you chose, 4-24 hours when a card we hold is fitted, 5-10 working days when one has to be bought in, and 10-15 working days for a current-generation accelerator on allocation.
• No contract: one month is the default term.
• When it is the wrong fit: if every question will go to a hosted model and nothing runs on a card, you do not need a GPU; any dedicated server will do, as long as it is not on a line without AVX2: the Atom, Opteron and Xeon E5-2600 or E5-4600 v1/v2 machines.

Questions people ask

Should I use the Desktop app instead?

If you are the only person using it, the project itself suggests the Desktop app. The Docker edition is for a shared instance: logins, roles, per-workspace access and embeddable chat widgets only exist there.

Do I really need a graphics card?

Not for AnythingLLM itself. It can send questions to a hosted model and turn documents into search vectors on the processor. You need the card when the model runs on your own server, either in the built-in engine or in something like Ollama next to it.

Can people just open the server's IP on port 3001?

Not with this setup, and that is deliberate. Port 3001 only answers on the server itself, so your colleagues use the HTTPS address Caddy serves. The only direct way to 3001 is the SSH tunnel you open yourself.

What happens to my documents if I delete the container?

Nothing. They live in /var/lib/anythingllm on the server, not in the container. The docs warn that without that folder mounted, everything is lost when the container restarts.

Does AnythingLLM send data anywhere?

It sends anonymous usage telemetry unless you opt out; the setup above sets DISABLE_TELEMETRY to true, and there is also a toggle under Privacy inside the app. The README adds that you will still see connections for model downloads and a few files fetched from GitHub.

Can I run several copies of it for more users?

Not against the same data. The default database is SQLite, and the cloud guide advises against several containers reading and writing one instance; for that it points to its separate PostgreSQL image.