What you end up running
SGLang is an open-source serving engine from LMSYS. You point it at a model, it keeps that model loaded on your graphics card, and any program that knows how to talk to OpenAI can talk to it instead: chat completions, plain completions, embeddings and vision models. Its documentation describes it as built for many people asking at once, with a prefix cache it calls RadixAttention so that conversations sharing the same opening do not pay for it twice.
The result is a private endpoint for your team, product or agent, on hardware you control, with no per-token bill and no one else's prompts on the same card.
Is your graphics card new enough?
Probably, if it is an Ampere-generation card (A100, RTX 30) or newer, which is where the SGLang README's supported list starts; Turing cards (T4, RTX 20, Quadro RTX) meet the sm75 minimum of its attention backend, yet the list leaves them out, so treat them as best-effort. The SGLang install guide is written for Linux on an NVIDIA card plus a driver that supports CUDA 13, and it says its default attention backend, FlashInfer, "only supports sm75 and above". Compute capability is NVIDIA's version number for what a chip can do, and 7.5 (sm75) is Turing. The docs do not set a single hard floor for the whole engine; they do offer a Triton-based fallback for when FlashInfer gives trouble.
Putting that next to NVIDIA's own table is our advice rather than SGLang's: on our list, aim for GeForce RTX 30 through 50, the A100, H100, L4 and L40S, the workstation RTX A-cards, plus the RTX PRO 6000 Blackwell; the T4, Quadro RTX line and RTX 20 cards are best-effort; and skip the Pascal and Volta cards such as the P40 and V100, however generous their memory.
On AMD, the SGLang README names four Instinct parts, from the MI300X up to the MI355X. The MI50 and MI210 cards we fit are not named there, so treat them as untested for SGLang.
How much card memory will your model want?
Start with the weights: two bytes per parameter at 16-bit, one at 8-bit, 0.5 bytes at 4-bit. That puts a 7-8 billion parameter model at about 16, 8 or 5 GB, and a 70B at about 140, 70 or 38 GB. Those are floors, not budgets.
SGLang reserves a block of memory as soon as it starts, enough for the weights and a cache of the conversations in flight. The --mem-fraction-static setting controls how big that block is, so don't be alarmed when the card looks full on an idle server. If you run out, the troubleshooting page suggests, in this order: smaller --chunked-prefill-size (4096 or 2048) for long prompts, a lower --max-running-requests, or --mem-fraction-static at 0.8 or 0.7, which means fewer people served at once.
Quantised models are the usual way to fit a bigger model on a smaller card. SGLang's advice is to download one already quantised (AWQ, GPTQ-Marlin or FP8, for example) rather than quantising on the fly, and not to add --quantization when you do. Our GPU servers page lets you filter cards by how much memory they carry.
Getting it running with Docker
- Put the NVIDIA driver on first, and check that
nvidia-smisees the card. - Install Docker and NVIDIA's Container Toolkit, then let Docker use it (first two lines below).
- Write a small settings file with your model, a key, and the listening address, readable by root only.
- Start the official
lmsysorg/sglangimage with that file mounted, publishing the port on the machine itself only. - Follow
docker logs -f sglanguntil it reports it is ready, then send a test request.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo mkdir -p /etc/sglang /srv/hf-cache
sudo tee /etc/sglang/config.yaml >/dev/null <<'EOF'
model-path: Qwen/Qwen3-0.6B
host: 0.0.0.0
port: 30000
api-key: change-me-to-a-long-random-string
EOF
sudo chmod 600 /etc/sglang/config.yaml
docker run -d --name sglang --gpus all --ipc=host --shm-size 32g \
-p 127.0.0.1:30000:30000 \
-v /srv/hf-cache:/root/.cache/huggingface \
-v /etc/sglang/config.yaml:/etc/sglang/config.yaml:ro \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server --config /etc/sglang/config.yamlcurl http://localhost:30000/v1/chat/completions \
-H "Authorization: Bearer change-me-to-a-long-random-string" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello"}]}'A word on the settings file: SGLang's config example uses the flag names without their leading dashes, but it never shows api-key itself, so once it is running, send one request without the key and make sure it is refused. Prefer no Docker? The docs' other route is uv pip install --prerelease=allow sglang in a Python 3.11+ virtual environment, then sglang serve with the same model. Either way, once it works, swap latest for a specific release tag so an update never surprises you.
Only your apps get in
Left to its defaults, SGLang listens on 127.0.0.1, which only the machine itself can reach. Most examples in its docs change that to --host 0.0.0.0 and publish Docker's port with -p 30000:30000, and that combination opens the model to the whole internet. Docker's documentation explains why a ufw firewall rule won't save you: published container ports are redirected before ufw's rules are checked.
- Keep the
127.0.0.1:in front of the published port, as above, and put an HTTPS proxy (port 443) in front when other machines need access. This needs Docker Engine 28 or later. On earlier versions, Docker warns, machines sharing your local network segment could still get through to a port published on localhost. And it only covers IPv4 loopback; the IPv6 one is[::1]. - Always set
api-key: it is the key the server is set with, and its OpenAI-compatible API checks it as well, and clients pass it as their OpenAI key. Keep it in the settings file; typed as a flag it would be visible to anyone runningps, and saved to your shell history. - Set
--admin-api-keytoo: it protects the admin and control endpoints, like reloading weights, flushing the cache and the/server_infopage. - Serving an image model to the public? Add
--allowed-media-domains, or SGLang will fetch pictures from any address a user supplies.
When one card is not enough
Add --tp 2 (or tensor-parallel-size: 2 in the settings file) and SGLang splits the model across two cards in the same machine. If it complains that "peer access is not supported", add --enable-p2p-check. When each card can hold the whole model and you simply want more people served, --dp runs separate copies instead.
The biggest single card we fit has 96 GB. Beyond that you are looking at several cards in one machine, which we quote rather than configure online; more than one card or machine explains what each platform gives them.
SGLang, vLLM, llama.cpp or Ollama?
There is no official speed comparison to lean on: SGLang's documentation does not benchmark itself against vLLM, and we won't invent one. The honest test is your model, your prompts and your number of users on the same card with each engine. If you want the closest alternative, our guide to a vLLM server walks through its setup the same way.
If your model comes as a GGUF file, or you'd like part of it to run on the processor, a llama.cpp server fits better. If you want the fewest decisions, Ollama pulls and runs models with one command; SGLang even answers Ollama-style requests, so the Ollama command-line tool and Python library can use it as a backend. And if your team wants a chat window rather than an API, Open WebUI works with all three.
Why a server of your own suits it
SGLang wants the card to itself, and here it gets it: your OS drives the card directly, without a hypervisor, a MIG partition or time-slicing, on a physical machine only you rent. The driver and CUDA version are yours to choose and freeze, every core and disk is yours, and the server's own remote console (iLO or iDRAC) shows you the screen even if a driver update cuts the network. The port is unmetered, so downloading large checkpoints or serving answers all day adds nothing to the bill. Monthly, no contract.
How soon you start depends on the card: about 30 minutes once the order is approved when a machine already carries it, 4-24 hours when we fit a card we hold, 5-10 working days when one has to be bought in, and 10-15 for a current-generation accelerator on allocation.
Questions people ask
- Can I reach my SGLang server from my laptop?
- Not straight away, and that is deliberate: by default it only answers on the server itself. The safe ways in are an SSH tunnel to port 30000, or a reverse proxy with TLS in front of it. Set an API key before you do either.
- Which card should I rent for SGLang?
- An Ampere-generation or newer NVIDIA card whose memory covers your model plus its cache. As a rough guide, a 70B model at 4-bit has 38 GB of weights alone, and serving it to several people at once wants about half as much again on top, so by our reckoning it needs an 80 GB card or the 96 GB RTX PRO 6000 Blackwell; an 8B model fits comfortably on much smaller ones.
- Is SGLang quicker than vLLM?
- Nobody can tell you from the official documentation, because SGLang's docs contain no side-by-side figures. It depends on the model, how much your prompts overlap and how many people ask at once. Try both on the same machine with your real traffic.
- Do I need Docker to run SGLang?
- No. The documentation also offers a pip or uv install into a Python 3.11+ environment, and a source install. Docker is simply the route with the fewest moving parts, since the image already carries CUDA 13 and every dependency.
- Can my existing OpenAI code talk to it?
- Yes. Set your client's base URL to your server's
/v1address and use your SGLang key as the API key. Chat completions, completions, embeddings and vision requests are all covered by the OpenAI-compatible API.