Serve your models with Triton Inference Server on a GPU server nobody else touches

Is Triton the right server for your models?

Triton is the right choice when you have more than one model to serve, or models in different formats. Each model gets its own folder, and your apps call every one of them over HTTP or gRPC the same way.

If all you want is a chat model behind an OpenAI-style API, Triton is more machinery than you need. Our guides to running vLLM and the llama.cpp server are the shorter road.

Will your card work with today's Triton?

Check the card's age first. NVIDIA's current release notes (26.10) support CUDA compute capability 7.5 and above, which means these families, shown here with the cards from our own list that belong to them:

• Turing: Tesla T4, the RTX 2070 and 2080 cards, Quadro RTX 4000, 5000, 6000 and 8000.
• Ampere: RTX A2000, A4000, A5000 and A6000, A40, A100 in both sizes, and the RTX 3070 to 3090.
• Ada Lovelace: RTX 4070, RTX 4090D, L4, L40S, RTX 6000 Ada and RTX 4500 Ada.
• Hopper and Blackwell: the H100 80GB, RTX 5090, RTX 5080 and RTX PRO 6000 Blackwell.
• Too old for current images: V100 and TITAN V (Volta), every Pascal card such as the Tesla P4, P40 and P100 or the GTX 10-series, and the Maxwell, Kepler and Fermi parts such as the Quadro 3000M. AMD Instinct and Tenstorrent cards are not NVIDIA devices; Triton would use only the processor there.

On a machine you already have, nvidia-smi --query-gpu=index,compute_cap --format=csv,noheader shows the number for each card. NVIDIA's frameworks support matrix lists CUDA 13.4.1 inside the 26.09 image used below, so the driver you install must be new enough for CUDA 13.

How big a server do you need?

Start from card memory. Unless you tell it otherwise, Triton loads every model in its folder when it starts (model_management.md), so all of them have to fit on the card together, and the runtime needs space of its own. Then the rest of the machine.

• Card memory: the total of your models, plus a copy for every extra instance you ask for. That sum is our rule of thumb, not NVIDIA's; the metric nv_gpu_memory_used_bytes tells you the truth once it runs.
• Processor cores: anything not on the card runs here, such as Python pre-processing, models you deliberately place on the CPU, or decoding images.
• Memory and shared memory: Python models talk to Triton through shared memory; the Python backend's README runs its container with --shm-size=1g, which is why the start script below does too.
• Disk: the Triton image plus your model folder, on drives inside the machine.

The server that fits

Our GPU servers hand the whole card to your own operating system. There is no hypervisor, no MIG partition and no time-slicing, so no second tenant's batch job moves your response times. You pin your own driver and CUDA version, so you choose the day you meet each monthly image's minimum driver.

Nothing else on the box is shared either. You get each core, the full memory, every drive and root, plus an HPE iLO or Dell iDRAC built into the server: a remote screen, virtual media and power buttons that keep working if a bad driver cuts the network. Nothing counts the bytes on the port, so downloads cost nothing extra. Want a second card, or more, inside one server? That is quoted rather than configured, and our multi-card servers page explains it.

How soon you can start depends on the card. Once the order is approved, a machine that already carries the card is ready in about 30 minutes; fitting a card we hold takes 4 to 24 hours; one that has to be bought in takes 5 to 10 working days, and a current-generation accelerator on allocation 10 to 15. All five cities are on the locations page.

Up and running in an afternoon

1. Install Ubuntu 24.04 on the server, then the NVIDIA driver through Ubuntu's package manager (the route NVIDIA's toolkit guide recommends), then Docker Engine using the apt repository Docker publishes. Reboot once the driver is in.
2. Paste the first block as a user with sudo. It switches the firewall on with only SSH allowed, checks that every card is 7.5 or newer, installs the NVIDIA Container Toolkit (its current release, where NVIDIA's guide pins one version) and runs NVIDIA's own test container.
3. Only once the first block has printed TOOLKIT OK, paste the second block. It saves the start command in /opt/triton/start.sh, fetches NVIDIA's densenet image classifier as a first model, and launches Triton under the unprivileged account the image ships with, reading the model folder read-only and listening only on 127.0.0.1.
4. Ask Triton what it loaded: curl -s 127.0.0.1:8000/v2/models/densenet_onnx should describe the model, with an input called data_0 and an output called fc6_1.

set -o pipefail   # for this shell: a failed download can no longer pass as success
sudo sed -i 's/^IPV6=.*/IPV6=yes/' /etc/default/ufw
sudo ufw allow OpenSSH && sudo ufw --force enable
low=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | sort -V | head -n 1)
if [ -n "$low" ] && [ "$(printf '7.5\n%s\n' "$low" | sort -V | head -n 1)" = 7.5 ] && command -v docker >/dev/null; then
  K=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
  curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --batch --yes --dearmor -o $K \
  && curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
     | sed "s#deb https://#deb [signed-by=$K] https://#g" \
     | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list >/dev/null \
  && sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit \
  && sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker \
  && sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi \
  && echo "TOOLKIT OK" \
  || echo "STOP - the toolkit install or the GPU test did not finish"
else echo "STOP - a card is below compute capability 7.5, or nvidia-smi or Docker is missing"; fi

R=/opt/triton/repository/densenet_onnx
sudo install -d -m 755 $R/1
sudo tee /opt/triton/start.sh >/dev/null <<'EOF'
TAG=26.09
docker run -d --name triton --restart unless-stopped --gpus all --net host --shm-size=1g \
  --user triton-server -v /opt/triton/repository:/models:ro \
  nvcr.io/nvidia/tritonserver:$TAG-py3 tritonserver --model-repository=/models \
  --model-control-mode=none --http-address=127.0.0.1 --grpc-address=127.0.0.1 \
  --metrics-address=127.0.0.1
EOF
U=https://raw.githubusercontent.com/triton-inference-server/server/r26.09/docs/examples/model_repository/densenet_onnx
sudo wget -qO $R/config.pbtxt $U/config.pbtxt \
  && sudo wget -qO $R/densenet_labels.txt $U/densenet_labels.txt \
  && sudo wget -qO $R/1/model.onnx https://github.com/onnx/models/raw/main/validated/vision/classification/densenet-121/model/densenet-7.onnx \
  && sudo bash /opt/triton/start.sh && n=0 \
  && until curl -fs 127.0.0.1:8000/v2/health/ready; do n=$((n+1)); [ $n -ge 60 ] && break; sleep 5; done \
  && curl -fs 127.0.0.1:8000/v2/health/ready && echo "Triton is up" \
  || { echo "STOP - Triton did not come up; recent log below"; sudo docker logs --tail 30 triton; }

If the second block stops, read the log lines it prints, fix the cause, clear the half-started container with sudo docker rm -f triton and paste the block again; the downloads simply overwrite. Pasting it while Triton already runs halts where it would create the container and prints STOP. That is expected: printf '%s\n' "$(curl -s -w '%{http_code}' -o /dev/null 127.0.0.1:8000/v2/health/ready)" printing 200 means your running copy is fine.

The container shares the host's network rather than publishing ports through Docker, so the firewall covers Triton like any program, and the 127.0.0.1 addresses add a second fence.

Getting more out of each card

Two settings in a model's config.pbtxt do most of the work. Dynamic batching runs requests that arrive close together as one batch, which usually means more answers per second. Instance groups decide how many copies of a model run and on which cards. Batching needs a model whose batch sits in its first dimension and a max_batch_size of at least 1; the densenet example has 0, so try this on a model of your own, for example an ONNX file saved as /opt/triton/repository/tagger/1/model.onnx:

# /opt/triton/repository/tagger/config.pbtxt
# ONNX inputs and outputs are filled in by Triton
platform : "onnxruntime_onnx"
max_batch_size : 8
dynamic_batching { }
instance_group [
  {
    count : 2
    kind : KIND_GPU
  }
]

That asks for two copies on every card with the batcher's defaults, NVIDIA's suggested starting point before tuning. Triton ignores folder changes while it runs, so restart it with sudo docker restart triton, then load-test from NVIDIA's client image with T=26.09; sudo docker run --net host --rm nvcr.io/nvidia/tritonserver:$T-py3-sdk perf_analyzer -m tagger --concurrency-range 2:16:2. Note that copies are whole models: instance groups never split one big model across two cards.

Keeping it private

• Triton has no usernames or passwords. NVIDIA's guide to secure deployment assumes a gateway or proxy in front that checks who is calling and encrypts the traffic.
• To use it from your own computer, open an SSH tunnel such as ssh -N -L 18000:127.0.0.1:8000 -L 18001:127.0.0.1:8001 -L 18002:127.0.0.1:8002 you@server, then point your client at localhost port 18000 (HTTP) or 18001 (gRPC), and your browser or Prometheus at 18002 for metrics.
• When your apps live on other servers, put a proxy with a real login (a token or client certificates over TLS) on this machine and allow only those servers' addresses through the firewall. To have Triton check client certificates itself on gRPC, edit /opt/triton/start.sh: point --grpc-address at the server's own address, append the TLS flags (--grpc-use-ssl-mutual=true, then --grpc-root-cert, --grpc-server-key, --grpc-server-cert), and let only your app servers reach port 8001 through ufw. Keep the certificates in /opt/triton/tls, made with sudo install -d -m 755 /opt/triton/tls && sudo install -o 1000 -g 1000 -m 600 /dev/null /opt/triton/tls/server.key and only then fill in the key; add -v /opt/triton/tls:/tls:ro to start.sh and give the three flags /tls/... paths. Backups of /opt/triton then contain that key, so keep them as private as the key.
• Leave model loading switched off (--model-control-mode=none). NVIDIA warns that loading models while the server runs could end with someone running code of their choosing, and that some backends, Python among them, run code from the model folder without a sandbox. Only put models you trust in /opt/triton/repository.
• Watch it through the metrics page at 127.0.0.1:8002/metrics. It is plain Prometheus text that you fetch, nothing is sent anywhere; nv_inference_queue_duration_us growing fast means requests are waiting for a free copy of the model.

Looking after it

Updating. NVIDIA publishes a new image each month, named by year and month. Wait until NGC actually lists the new tag, and read that month's release notes for the driver it needs. Then fetch the image first and swap afterwards, putting the new tag where YY.MM stands: T=YY.MM; sudo docker image pull nvcr.io/nvidia/tritonserver:$T-py3 && sudo sed -i -e "s/^TAG=.*/TAG=$T/" /opt/triton/start.sh && sudo docker stop triton && sudo docker rm triton && sudo bash /opt/triton/start.sh. If something breaks, run the same line with the previous tag, which Docker keeps locally until you delete it.

One bad model stops the server. By default Triton exits when anything fails during start-up, so a typo in one config.pbtxt keeps all your models offline until it is fixed. After every change, look at sudo docker logs triton.

Backups. Everything that matters is in /opt/triton: your models and the start script. sudo tar -czf /root/triton-backup-$(date -u +%F-%H%M%S).tar.gz -C /opt triton packs it; run sudo tar -tzf on the new file to prove it reads back, then copy it to another machine yourself.

Bringing a backup back. Put the archive's path in a, then run `O=/opt/triton; sudo test -f "$a" && [ ! -e $O.before-restore ] && [ ! -e $O.failed-restore ] && sudo tar --list --gzip --file "$a" >/dev/null && { sudo docker stop triton; sudo docker rm triton; true; } && sudo mv $O $O.before-restore && sudo test -f $O.before-restore/start.sh && sudo tar -xzf "$a" -C /opt && sudo bash $O/start.sh || echo "STOP - nothing after the failed step ran". It checks the archive, sets your folder aside and confirms the move before unpacking, and refuses if a folder from an earlier restore remains. Give Triton a few minutes, then printf '%s\n' "$(curl -s -w '%{http_code}' -o /dev/null 127.0.0.1:8000/v2/health/ready)"` should print 200.

If the restored copy will not come up, go back with O=/opt/triton; [ -d $O.before-restore ] && [ ! -e $O.failed-restore ] && { sudo docker stop triton; sudo docker rm triton; [ ! -e $O ] || sudo mv $O $O.failed-restore; } && sudo mv $O.before-restore $O && sudo bash $O/start.sh. If the restore stopped before anything moved, sudo bash /opt/triton/start.sh restarts your untouched folder. Once the restored copy works, sudo rm -rf /opt/triton.before-restore; after a rollback, keep what you need from /opt/triton.failed-restore, then delete it. Either folder left behind stops the next restore.

Licence. NVIDIA releases Triton's source under BSD-3-Clause: you may use and change it freely, and any copy you hand on must carry NVIDIA's copyright line and the licence wording, with no obligation to share source. The NGC image also contains other software under its own terms. That is how we read it; it is not legal advice.

When to look elsewhere. A single server does not scale itself; for a fleet, NVIDIA's README links deployment examples for Kubernetes with Helm. A model bigger than our largest single card, 96 GB, needs a backend that splits it plus a quoted multi-card build.

Questions people ask

Which ports does Triton use?

HTTP on 8000, gRPC on 8001 and Prometheus metrics on 8002 by default. In the setup above all three listen on 127.0.0.1 only, so the internet sees none of them until you put a proxy in front or deliberately change those addresses in /opt/triton/start.sh.

Which model files can I drop into the folder?

TensorRT plans named model.plan, ONNX files named model.onnx, TorchScript files named model.pt, OpenVINO models and Python scripts named model.py, each inside a numbered version folder. Most TensorRT, ONNX and OpenVINO models need no config.pbtxt; PyTorch and most others do.

Can I add a second card later?

Triton needs no change for it: it places one copy of each model on every card it sees, so once a second card is in, restarting the container is all it takes. A machine with more than one card is quoted rather than configured, so ask for that build.