Your own vLLM server, answering many users at once

Will my graphics card work?

Only if it is new enough, and that catches people out. vLLM supports NVIDIA cards of compute capability 7.5 and up, a number NVIDIA gives every generation. Plenty of older cards have lots of memory but sit below that line.

• Works: the Tesla T4; GeForce RTX cards from the 20 series onward (30, 40 and 50 too); the Quadro RTX 4000 and 5000; the RTX A2000, A4000 and PRO 6000; and datacenter cards like the A100, L4 and H100.
• Does not: Tesla M60, P4, P40 and P100, the GTX 10 series and the TITAN V, all from generations before Turing.
• AMD: only the cards vLLM names, through ROCm 6.3 or later: Instinct MI200 (the MI210, for one), MI300 and MI350, plus the Radeon 7900-series and 9000-series graphics cards. The older Instinct MI50 is not among them.

vLLM runs on Linux only. Its guide sends Windows users to WSL, so on a rented server simply choose Linux.

How much card memory will your model need?

Two things share the card. First the model itself: roughly 2 GB per billion parameters at full 16-bit precision, or about a quarter of that for a 4-bit version. Then the KV cache, which is where vLLM keeps track of every conversation it is answering. The bigger that space, the more users it can serve at once and the longer their chats can be.

Leave generous room on top for the cache. By default vLLM takes 92% of the card's memory when it starts and keeps it, so do not plan to run anything else on the same card.

Up and running in an evening

1. Order a GPU server with Ubuntu and install NVIDIA's driver. nvidia-smi should list your card.
2. Install vLLM into its own Python environment. The guide's recommended command is uv pip install vllm --torch-backend=auto, followed by vllm download-kernels.
3. Choose a model from Hugging Face that fits your card, and start it with vllm serve, keeping it on the server's own address (127.0.0.1) for now.
4. Point your app at http://localhost:8000/v1 with the API key you set, and send a test question.

# a 4-bit 7B model leaves room for chats on a 12-16 GB card
export VLLM_API_KEY="pick-a-long-random-string"
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --host 127.0.0.1 --port 8000 --max-model-len 8192

Prefer Docker? vLLM publishes an official image, vllm/vllm-openai, which starts the same server. Give it the graphics card, your Hugging Face cache folder and --ipc=host, which the docs ask for because the parts of vLLM share data through memory. Publish the port to the server itself only, with -p 127.0.0.1:8000:8000: a plain -p 8000:8000 opens it to the internet, and Docker's ports skip ufw.

Locking it down

Setting an API key is not the whole job. vLLM's documentation says the key only guards addresses starting with /v1, /v2 and /inference, while other parts of the same server, such as /invocations, answer anyone. Its own advice is not to rely on the key alone.

• Keep vLLM listening on 127.0.0.1 only.
• Put a web server such as nginx or Caddy in front with HTTPS, passing on only the /v1 addresses your apps use.
• Check keys and limit how many requests each one can make at that front door.
• Close every other port in the firewall except SSH and HTTPS (443), plus 80 if your proxy uses it to get its certificate.

The server that fits

Because vLLM wants the whole card, it suits a whole card. On our GPU servers your operating system gets the graphics card directly. There is no shared slice and no virtual GPU, and nobody else's job uses its memory. The list lets you filter by how much card memory you need, and shows which machine each card goes in.

You pay by the month, not by the hour, and the port is unmetered both ways, so a busy day of answers never turns into a traffic bill. Need more memory than one card has? vLLM can split a model across several cards in one machine, and our multi-GPU servers are built for that. To weigh up a simpler setup first, our Ollama guide covers the one-user case.

When something simpler will do

If it is just you, or a handful of colleagues asking now and then, vLLM is more machinery than you need; Ollama or llama.cpp is easier to live with. vLLM is the right call when an app or a team sends a steady stream of requests and you want each one answered quickly.

Questions people ask

Can I run vLLM on an older Tesla P40?

No. The P40 is a Pascal card, below the compute capability 7.5 that vLLM requires. It can still run models through llama.cpp or Ollama.

Will my existing OpenAI code work with vLLM?

Mostly, yes. vLLM offers OpenAI-style chat completions, completions, responses and embeddings. Change the base address to your server and use your own key.

Why does vLLM take nearly all my card's memory?

It reserves 92% by default so it has room to track many conversations at once. You can lower that with --gpu-memory-utilization, but it will then serve fewer requests in parallel.

How big a model fits on a 24 GB card?

A 16-bit 8B model fits with room for chats, and a 13-14B model in 4-bit (about 8 GB) leaves plenty of room for many users. For a 30B model in 4-bit, look at a 32 GB or 48 GB card. Always leave space for the KV cache, not just the weights.

Do I need Windows support?

vLLM is Linux-only; on Windows it runs through WSL. For a server, install Ubuntu or another Linux and save yourself the extra layer.

Is the API key enough to keep strangers out?

Not by itself. It protects only part of the server's addresses, so put a proxy with HTTPS and its own checks in front, and keep vLLM itself on 127.0.0.1.