Workload · Running a private coding assistant

Tabby self-hosted coding assistant: code completion on a server you control

Tabby is an open-source coding assistant you host yourself: one Docker container serves code completion and chat to VS Code, IntelliJ-based IDEs and Vim, and the code your team types is answered by models running on a machine you control. What it needs is an NVIDIA card sized to the two models you choose, about an evening of setup, and a little care over who gets to register first.

The short version
  1. Tabby's docs suggest a T4, 10 or 20 Series card for 1B-3B completion models
  2. 7B-13B models: V100, A100, 30 or 40 Series, per the same docs
  3. The first account registered becomes the owner, so register before you open it up
  4. Usage statistics to the Tabby team switch off with one environment variable

What Tabby asks of the hardware

Tabby needs a graphics card more than anything else, and the size of card depends on the completion model you run. Tabby's model registry gives the guidance by model size rather than by gigabytes, so the table below repeats it and puts the cards we fit beside each bracket.

Completion model sizeTabby's suggested GPUsCards on our list in those families
1B to 3B (StarCoder-1B, Qwen2.5-Coder-1.5B, CodeGemma-2B)NVIDIA T4, 10 Series or 20 SeriesTesla T4, GTX 1070, 1070 Ti, 1080, 1080 Ti, RTX 2070, 2070 Super, 2080, 2080 Super and 2080 Ti
7B to 13B (Qwen2.5-Coder-7B, StarCoder2-7B, CodeLlama-13B)NVIDIA V100, A100, 30 Series or 40 SeriesTesla V100 32GB, A100 40GB and 80GB, RTX 3070 to 3090, RTX 4070, RTX 4090D

Tabby downloads registry models at 8-bit (q8_0), so a 7B model is about 8 GB of weights and a 13B about 13 GB, before the chat model. The 8 to 12 GB cards in that row are too small for a 7B pair; choose a 24 GB card or larger.

The chat model is chosen separately with --chat-model, and Tabby's docs only ask that it have at least 1B parameters. Both models load on the same card, so the pair has to fit together, with room left for the work of answering. The registry publishes no per-model memory figure; our guide to sizing a language model has the arithmetic for weights at 16, 8 and 4-bit if you want to estimate it before ordering.

The docs give no hardware advice for the larger entries, such as Codestral-22B or Qwen2.5-Coder-14B. If you want one of those, size the card from the model's weights and leave generous headroom.

Why a whole machine suits a team's assistant

Code completion is only useful when it answers while you are still typing, so the card should be yours alone. On our GPU servers it is: the card sits in a physical server rented to one account and is handed to your operating system directly. There is no MIG partition, no vGPU slice and no time-slicing, so nobody else's batch job slows your suggestions in the middle of the afternoon.

You install and pin your own NVIDIA driver and CUDA release, with root on the whole machine. Rack servers come with an HPE iLO or Dell iDRAC console, so if a driver update takes the network down you can still see the screen and fix it. The port is unmetered, which matters the day you pull a 7B model or index a large repository: nothing counts the bytes. Billing is monthly with no contract; a card that has to be bought in for you is paid three months up front.

The server that fits

Every card we fit is listed on the GPU dedicated servers page, priced as a separate line from the machine it goes in, so you can compare a 20 Series card for a small team's 1B model against an RTX 3090, V100 or A100 for a 7B pair. Consumer GeForce cards are the honest choice for an internal tool where a rare wrong bit means a retried suggestion; the data-centre accelerators and most of the professional cards add ECC memory if that matters to you.

Delivery depends on the card: about 30 minutes once the order is approved if a machine already carries it, 4 to 24 hours when we fit a card we hold, and 5 to 10 working days when one has to be bought in. Instant servers shows what is racked right now, and the full dedicated server catalogue covers the machine half.

Up and running in an evening

  1. Install Ubuntu from the order form, then the NVIDIA driver and Docker. Confirm nvidia-smi lists your card before going further.
  2. Install the NVIDIA Container Toolkit by following NVIDIA's own install guide; Tabby's docs require it for CUDA in Docker.
  3. Start Tabby with the command below. It is the official one with two changes: the port is published on the loopback address only, so nobody on the internet can reach the sign-up page before you do, and usage collection is switched off.
  4. From your own computer, open a tunnel with ssh -L 8080:127.0.0.1:8080 you@your-server and browse to http://localhost:8080.
  5. Register straight away. Tabby makes the first account registered after deployment the admin, with the owner role.
docker run -d \
  --name tabby \
  --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v $HOME/.tabby:/data \
  -e TABBY_DISABLE_USAGE_COLLECTION=1 \
  registry.tabbyml.com/tabbyml/tabby \
  serve \
  --model StarCoder-1B \
  --chat-model Qwen2-1.5B-Instruct \
  --device cuda

Watch the first start with docker logs -f tabby; the models download into $HOME/.tabby on the host, so they survive the container being replaced. The official Docker Compose example sets restart: always; with docker run, add Docker's --restart always flag if you want Tabby back after a reboot.

Keep the 127.0.0.1: part even after setup: a reverse proxy on the same machine reaches Tabby there. Without an address, Docker publishes the port on every address of the host, plain HTTP included.

Connecting VS Code, IntelliJ and Vim

Each editor needs two things: the server's address and your personal token. After signing in, Tabby's home page shows the credentials for connecting IDE extensions.

  1. Install the Tabby extension for your editor: VS Code and VSCodium, the IntelliJ Platform, or Vim and Neovim.
  2. In VS Code, run Tabby: Connect to Server... from the command palette and paste the endpoint URL and token.
  3. For settings shared by every editor on one computer, use the agent's file at ~/.tabby-client/agent/config.toml, shown below. The extension's own anonymous usage tracking is switched off there too.
[server]
endpoint = "http://localhost:8080"
token = "your-token-here"

[anonymousUsageTracking]
disable = true

Teammates are invited from the Users & Groups page: by email, or by copying the sign-up link from the Pending Invites list. Emailed invites need mail delivery set up; their links use the External URL from General Settings.

Keeping it private and kept up

  • Who can sign up. Authentication Domains let anyone with a matching email address register without an invitation. Add only your own company's domain, or none.
  • What leaves the server. With TABBY_DISABLE_USAGE_COLLECTION=1 set, the server stops sending its usage report to the Tabby team. The fields that report lists are about the model, device and version, not your files.
  • Reaching it as a team. Put an HTTPS reverse proxy on the same machine in front of the loopback port, then set the External URL in General Settings to that https:// address.
  • Your own code as context. Under Integrations, then Context Providers, add a Git repository; Tabby indexes it and uses it for completion, chat and search. Private GitHub or GitLab repositories need a personal access token.
  • Backups. Accounts and settings live in $HOME/.tabby/ee/db.sqlite. Install the SQLite command-line tool first (sudo apt install sqlite3), then open the database as root with sudo sqlite3 $HOME/.tabby/ee/db.sqlite, run .backup backup_database.db, and copy the backup off the server.
  • Upgrades. Back up first: Tabby does not support downgrading. Then pull the image you run with docker pull registry.tabbyml.com/tabbyml/tabby, and re-create the container with the same command. The models and database stay in the mounted folder.

Before you remove the old container during an upgrade, check that the backup file exists and is not empty. Removing a container leaves the $HOME/.tabby folder alone, but a failed new version with no backup leaves you nowhere to go back to.

When a rented GPU is the wrong answer

A single developer with an Apple Silicon Mac can run a 1B to 3B model locally, which Tabby's docs list as suitable, and needs no server at all. If you already run llama.cpp or vLLM, Tabby can use it instead of loading its own models: chat through [model.chat.http] with the OpenAI-compatible kind, and completion through [model.completion.http] with that server's own completion kind and a prompt template for the model, both in ~/.tabby/config.toml, so a second card may be unnecessary. And if your organisation forbids source code leaving the office, a server in a data centre, however private, may still be outside your rules: check before ordering.

Questions people ask

Can Tabby run without a graphics card?
Yes. Tabby publishes a CPU-only build, started without the --device cuda flag. Its hardware advice for both model size brackets names GPUs or Apple Silicon, though, so plan for a card if a team will use it every day.
Which models should a small team begin with?
Start with the pair in Tabby's own example, StarCoder-1B for completion and Qwen2-1.5B-Instruct for chat, and see how much card memory is left. Tabby downloads registry models at 8-bit, so a 7B completion model is about 8 GB of weights before the chat model. With a 24 GB or larger card (RTX 3090, RTX 4090D, V100 32GB, A100) you can then move to a 7B completion model such as Qwen2.5-Coder-7B. Changing models means removing the container and starting it again with the same command and the new model names; the downloaded models and the database stay in $HOME/.tabby.
Does any of our code go to the Tabby team?
The server's usage report covers fields such as the model, device, CPU and version, and it stops when TABBY_DISABLE_USAGE_COLLECTION is set. The editor extensions have their own anonymous usage tracking, which you switch off in the agent's config file. Completions themselves come from the models on your own server.
How do I stop strangers from creating accounts?
Keep the port on the loopback address until you have registered, because the first account becomes the owner. After that, add members by invitation and leave Authentication Domains empty or set to your own domain only.
Is the GPU shared with other customers?
No. The card is in a physical server rented to one account and passed straight to your operating system, with no partition, virtual slice or scheduler between you and it. You choose the driver and CUDA version yourself.

Give your team its own coding assistant

Pick the card that fits your Tabby models, then run it on a whole machine with an unmetered port and no contract.

Choose a GPU server for Tabby

Read next