How much GPU memory does a large language model need?

The weights are the parameter count multiplied by the bytes per parameter: two bytes each at 16-bit, one at 8-bit, half at 4-bit. A 7-8 billion parameter model is 16 GB of weights at 16-bit, 8 GB at 8-bit and 5 GB at 4-bit; a 70 billion parameter model is 140 GB, 70 GB and 38 GB. Those figures are the weights alone and are a floor rather than a requirement -- budget a quarter to a half again on top for the runtime, the activations and the key-value cache.

Why does my model load but fail to serve?

Because the weights are not the only thing in card memory. The runtime and its buffers are resident, and every token in a conversation leaves an entry in the key-value cache so it does not have to be recomputed. That cache grows with the context length and again with each request served concurrently, so a model whose weights only just fit will hold roughly one short conversation. Size the card at one and a quarter to one and a half times the weights.

Which inference runtime can I install?

Any of them. The machine arrives with a clean operating system and root, and there is no vendor inference service in front of it. vLLM is the usual answer for serving real traffic because continuous batching lets one card answer many requests at once. llama.cpp is the usual answer when memory is tight or the card is older. TGI and SGLang run unchanged. Ollama is the shortest path from an empty machine to a model answering on a port. You pin the driver, the CUDA or ROCm release and the framework to the versions your code was tested against.

How many tokens per second will I get?

Primcast does not publish a tokens-per-second figure, because every such number depends on the model, the quantisation, the runtime and its version, the batch size, the prompt length, the reply length and the concurrency, and changing any one moves it by a multiple. What governs it: reading the prompt is compute-bound and happens once per request; generating the reply is memory-bandwidth-bound because each token requires reading the weights; and batching amortises that reading across concurrent requests, which is what makes serving economical until the key-value cache runs out of room. Ask with the model, quantisation and expected concurrency for an answer based on what has actually been observed.

Is the GPU shared with anyone else?

No. One tenant per physical machine, and the card is passed through to your own operating system. There is no hypervisor, no MIG partition, no vGPU profile and no time-slicing scheduler, so nothing else runs on the card and inference latency does not move because of somebody else's batch job.

What if the model does not fit on one card?

Then it is a multi-card build, which is quoted rather than ordered from a checkout: how many cards fit depends on the physical width of the specific card and the power budget of the chassis. Two cards give you the sum of their memory only if the runtime can split the model across both, which every serious runtime supports at some cost in throughput. https://primcast.com/hpc covers what more than one card and more than one machine look like.

Can I run an AMD card, and does ROCm work?

Yes. The AMD Instinct parts are among the largest-memory cards fitted and are frequently the best value per gigabyte, but your stack has to support ROCm rather than CUDA. Most current inference runtimes do; some surrounding tooling still does not. Ask before ordering and it will be checked against your specific runtime rather than answered in general.

Language models on bare metal · Primcast LLC · since 2004

The model decides the memory. Nothing else on the order form does.

People arrive at this page believing the decision is which accelerator is fastest. It is not. The weights either fit in card memory or they do not — a card two gigabytes short does not run slower, it fails to load. So this page goes in the order the decision actually goes: how big is the model, what precision will you run it at, how much room does the conversation need, and only then which card clears the bar. The prices are one page away, deliberately.

Which card runs my model What I install on it

One tenant per machine. The card is passed through to your operating system, and the runtime, the driver and the framework versions are yours to choose and to pin.

Step one

What the weights actually weigh.

Parameters multiplied by bytes per parameter, and that is the whole calculation. Sixteen-bit weights are two bytes each, eight-bit are one, four-bit are a half. A seven-billion-parameter model at sixteen bits is fourteen gigabytes of weights; the same model at four bits is between three and four.

Quantising is what turns a machine that cannot hold a model into one that can. It costs quality, and how much depends on the model and on the method far more than on the bit count alone — a well-quantised four-bit model is usually closer to its full-precision self than people expect, and a badly quantised eight-bit one can be worse. If accuracy is the point of the exercise, test both before committing to hardware; the difference is cheaper to discover now than after delivery.

These figures are the weights and nothing else. They are the floor, not the requirement. The next section is the part that catches people out.

Weights only — the runtime, the activations and the conversation are extra, and are the subject of the next section.
Model	16-bit	8-bit	4-bit
7-8 billion parameters	16 GB	8 GB	5 GB
13-14 billion parameters	28 GB	14 GB	8 GB
30-34 billion parameters	68 GB	34 GB	19 GB
70 billion parameters	140 GB	70 GB	38 GB

Step two

Why a model that fits still will not serve.

This is the single most common way a GPU purchase goes wrong, and it goes wrong in the same way every time: somebody reads that their model is thirty-eight gigabytes, buys a forty-gigabyte card, loads it successfully, and then discovers it can hold roughly one short conversation.

The runtime is in there too

Loading a model is not the only thing occupying the card. The runtime itself, its buffers and the activations of whatever is being computed all live in the same memory as the weights.
The conversation is in there, and it grows

Every token already in a conversation leaves something behind in the key-value cache so it does not have to be recomputed. That cache grows with the length of the context and again with every request being served at the same time. A long context or a busy endpoint can want as much memory as a small model does.
So: a quarter to a half again

Budget at least a quarter more card memory than the weights, and half again if the context is long or the endpoint is busy. That is the rule the table in the next section applies for you — every card it names holds the weights one and a half times over.
And if it is close, say so before you buy

Tell us the model, the quantisation, the context length you need and roughly how many requests at once. We would rather talk you into a larger card now, or a smaller model, than take an order that disappoints in week one and comes back as a refund request.

Step three

Which card runs which model.

Read down to your model size and across to the precision you intend to run at. Each cell names the smallest card we fit that holds those weights with room left to serve, and links to the machine it goes in. Where the catalogue holds nothing that does it alone, the cell says so rather than leaving a blank.

The smallest card we fit that holds the weights with room left to serve — that is 1.5× the weights, the generous end of the quarter-to-a-half head-room rule below. No prices here on purpose: GPU dedicated servers holds every one.
If you want to run	at 16-bit	at 8-bit	at 4-bit
7-8 billion parameters	TESLA P40 / QUADRO P600024 GB16 GB of weights	TITAN V12 GB HBM28 GB of weights	TESLA P48 GB GDDR55 GB of weights
13-14 billion parameters	RTX A600048 GB GDDR6 ECC28 GB of weights	TESLA P40 / QUADRO P600024 GB14 GB of weights	TITAN V12 GB HBM28 GB of weights
30-34 billion parameters	More than one cardNothing we fit holds it alone68 GB of weights	Instinct MI21064 GB HBM2e34 GB of weights	Tesla V100 32GB32 GB HBM219 GB of weights
70 billion parameters	More than one cardNothing we fit holds it alone140 GB of weights	More than one cardNothing we fit holds it alone70 GB of weights	Instinct MI21064 GB HBM2e38 GB of weights

Three things the table cannot tell you

The smallest card that fits is not always the card you want. Memory decides whether a model runs; everything else about the card decides how fast it runs and how many people it can serve at once. The table optimises for the first because the first is a hard wall and the second is a preference. If throughput matters more than budget, go a step up the list below.
Older cards do not speak the newer number formats. Several parts in our catalogue predate FP8 and bfloat16, so a runtime that expects them will fall back to something slower or refuse. That is a question worth asking before ordering, and one we will answer for a specific card and a specific runtime rather than in general.
An AMD card runs ROCm, not CUDA. The Instinct parts are excellent value per gigabyte and they are the answer in more than one cell above, but your stack has to support ROCm. Most current runtimes do; some tooling around them still does not. Check yours, or ask and we will check it with you.

When the answer is “more than one card”, that is a build we quote rather than a checkout option, and multi-GPU and multi-node is the page that explains what it looks like and what limits it. What each of these cards costs per month, and what the machine under it costs, is GPU dedicated servers — every figure there is attached to the exact machine it belongs to, which is why this page does not repeat any of them.

By memory, largest first

Every card we fit that publishes a memory figure.

The same catalogue the GPU page prices, sorted the other way round: that page starts at the cheapest complete machine because it is answering “what does this cost”, and this one starts at the largest card because it is answering “what is the biggest thing I can run”. A card whose memory the manufacturer does not publish is left out here rather than shown with a blank, because a row that cannot be compared is no use in a comparison.

Several of these appear on more than one chassis, sometimes at different prices, and the bus generation of the machine underneath changes what a modern card can actually do — both of which are on GPU dedicated servers. What is standing racked and ready this minute is a different question again, answered by instant servers.

What you install on it

The runtime is yours, and that is the point of bare metal.

The machine arrives with a clean operating system and root. There is no vendor inference service you have to go through, no managed runtime that upgrades itself the week before a deadline, and no API in front of your own model. You install what you want and pin it to the version your code was tested against.

vLLM

The usual answer for serving a model to real traffic. Continuous batching and paged attention are what let one card answer many requests at once instead of one at a time, which is normally the difference between a demo and a service. Wants a reasonably modern card.

llama.cpp

The usual answer when memory is tight or the card is older. Its quantised formats are the reason a large model fits on a small card at all, and it will happily use processor and card together when the weights do not quite fit. Slower per request, far more forgiving.

TGI and SGLang

Production servers in the same family as vLLM, with different strengths around structured output, prefix reuse and multi-model serving. If you already have one in your stack, it runs here unchanged.

Ollama

The shortest path from an empty machine to a model answering on a port. Worth starting with even if you end up somewhere else, because it will tell you in about ten minutes whether your sizing was right.

Whichever you choose, the driver, the CUDA or ROCm release and the framework build are yours to pin and nothing upgrades underneath you. If you would rather the driver were installed before handover, say so on the order and name the version.

How fast will it be

We do not publish a tokens-per-second figure, and here is why.

Every such number depends on the model, the quantisation, the runtime and its version, the batch size, the length of the prompt, the length of the reply and how many requests are in flight. Change any one and the figure moves by a multiple. A headline number that does not carry all of that is a number chosen to look good, and we would be quoted on it.

What we will tell you is what governs it, so you can reason about your own case:

Reading the prompt is compute-bound. It happens once per request, in parallel across the whole prompt, and it is where a card’s arithmetic throughput and its support for the newer number formats show up.
Generating the reply is memory-bandwidth-bound. Each token requires reading the weights, so for a single request the ceiling is roughly how fast the card can stream its own memory. This is why a card with slower memory and more of it can lose to a smaller, faster one on latency while winning on whether the model fits at all.
Batching is what makes it economic. Serving several requests together amortises that weight-reading across all of them, which is why total throughput climbs steeply with concurrency while per-request latency barely moves — until the key-value cache runs out of room, at which point everything degrades at once. Head-room again.

Ask for a real answer instead

Tell us the model, the quantisation you intend, the context length and roughly how many concurrent requests you expect. If we have run something comparable we will say what we saw and on what. If we have not, we will say that too.

It is a better basis for a purchase than a benchmark chart, and it takes about a minute.

Ordinary support hours are all of them — telephone, chat and tickets, every day of the year. Support has the response times in writing.

Beyond serving

Fine-tuning needs considerably more room than serving does.

Everything above sizes a model for inference: weights in, tokens out. Training or fine-tuning the same model is a different memory problem, and the difference is not small.

Why it costs more memory

Serving needs the weights. Training needs the weights, the gradients and the optimiser’s own state for every parameter it updates, all resident at once, plus the activations it has to keep in order to work backwards through the network. A model that serves comfortably on one card can be several times too large to fully fine-tune on the same card.

Which is why most people do not do it fully

Parameter-efficient methods — adapters, low-rank updates and their quantised variants — update a small fraction of the parameters and need a correspondingly small fraction of the extra memory. For most tasks that people call fine-tuning, this is both the practical choice and the one that fits on hardware you can afford to keep.

What to tell us

The model, the method, the sequence length and the dataset size. We will not publish a multiplier here because the honest one depends on all four, and a made-up figure that under-sizes your machine is worse for both of us than a conversation. If the answer is more than one card, multi-GPU and multi-node explains what that build looks like.

The bus matters here in a way it does not for serving

A training loop streams batches across the link between processor and card continuously, so the PCIe generation of the machine underneath is a real constraint rather than a specification. Inference on a resident model barely touches it. Which platform gives a card which link is published on GPU dedicated servers and compared across platforms on multi-GPU and multi-node.

Where to go next

The pages either side of this one.

This page owns one question: what it takes to run a model. Everything adjacent has its own page, and this is all of them.

Asked before

The eight questions this page exists to answer.

Taken from what people actually ask us in chat and on tickets, in roughly the order they ask it. The first two are the ones that decide whether a purchase works.

How much GPU memory does a large language model need?: The weights are the parameter count multiplied by the bytes per parameter: two bytes each at 16-bit, one at 8-bit, half at 4-bit. A 7–8 billion parameter model is 16 GB of weights at 16-bit, 8 GB at 8-bit and 5 GB at 4-bit; a 70 billion parameter model is 140 GB, 70 GB and 38 GB. Those figures are the weights alone and are a floor rather than a requirement — budget a quarter to a half again on top for the runtime, the activations and the key-value cache.
Why does my model load but fail to serve?: Because the weights are not the only thing in card memory. The runtime and its buffers are resident, and every token in a conversation leaves an entry in the key-value cache so it does not have to be recomputed. That cache grows with the context length and again with each request served concurrently, so a model whose weights only just fit will hold roughly one short conversation. Size the card at one and a quarter to one and a half times the weights.
Which inference runtime can I install?: Any of them. The machine arrives with a clean operating system and root, and there is no vendor inference service in front of it. vLLM is the usual answer for serving real traffic because continuous batching lets one card answer many requests at once. llama.cpp is the usual answer when memory is tight or the card is older. TGI and SGLang run unchanged. Ollama is the shortest path from an empty machine to a model answering on a port. You pin the driver, the CUDA or ROCm release and the framework to the versions your code was tested against.
How many tokens per second will I get?: We do not publish a tokens-per-second figure, because every such number depends on the model, the quantisation, the runtime and its version, the batch size, the prompt length, the reply length and the concurrency, and changing any one moves it by a multiple. What governs it: reading the prompt is compute-bound and happens once per request; generating the reply is memory-bandwidth-bound because each token requires reading the weights; and batching amortises that reading across concurrent requests, which is what makes serving economical until the key-value cache runs out of room. Ask with the model, quantisation and expected concurrency for an answer based on what we have actually observed.
Is the GPU shared with anyone else?: No. One tenant per physical machine, and the card is passed through to your own operating system. There is no hypervisor, no MIG partition, no vGPU profile and no time-slicing scheduler, so nothing else runs on the card and inference latency does not move because of somebody else’s batch job.
What if the model does not fit on one card?: Then it is a multi-card build, which is quoted rather than ordered from a checkout: how many cards fit depends on the physical width of the specific card and the power budget of the chassis. Two cards give you the sum of their memory only if the runtime can split the model across both, which every serious runtime supports at some cost in throughput. More than one card or machine covers what that build looks like.
Does fine-tuning need more memory than serving?: Considerably more. Serving needs the weights; training needs the weights, the gradients and the optimiser state for every parameter being updated, all resident at once, plus the activations kept for the backward pass. A model that serves comfortably on one card can be several times too large to fully fine-tune on it. Parameter-efficient methods update a small fraction of the parameters and need a correspondingly small fraction of the extra memory, which is why most fine-tuning here uses one.
Can I run an AMD card, and does ROCm work?: Yes. The AMD Instinct parts are among the largest-memory cards we fit and are frequently the best value per gigabyte, but your stack has to support ROCm rather than CUDA. Most current inference runtimes do; some surrounding tooling still does not. Ask before ordering and we will check it against your specific runtime rather than answer in general.

For the record

Everything on this page, as figures.

Card memory is the manufacturers’ published figure; which cards exist is read from the order catalogue when this page was served. No monthly price appears here on purpose — GPU dedicated servers holds every one, against the exact machine it belongs to.

What decides whether a model runs: Card memory, and almost nothing else on the order form. The weights either fit or they do not — a card a couple of gigabytes short does not run slower, it fails to load. Clock speed and core count decide how fast it is once it fits.
How much memory a model needs: Parameters multiplied by bytes per parameter: two bytes each at 16-bit, one at 8-bit, half at 4-bit. That is the WEIGHTS. The runtime, the activations and the key-value cache for the conversation live in the same memory, so budget a quarter to a half again on top before choosing a card.
Largest model on a single card here: 70 billion parameters at 4-bit, with room left to serve. Past that the answer is more than one card in one machine, which is a build we quote rather than a checkout option — multi-GPU and multi-node is the page for it.
Biggest card we fit: RTX PRO 6000 Blackwell 96GB, 96 GB GDDR7 ECC. Card memory figures are the manufacturers' published ones; a card whose figure we could not source is left out of the comparison rather than guessed at.
Which runtime: Yours. The machine arrives with a clean operating system and root, and you install vLLM, llama.cpp, TGI, SGLang, Ollama or anything else, pinned to the version your code was tested against. Nothing upgrades underneath you, and there is no vendor runtime you have to go through.
Throughput: We publish no tokens-per-second figure, deliberately, because we have not measured one under conditions we would be willing to have quoted back at us. What governs it is the card's memory bandwidth while generating, its arithmetic while reading the prompt, and how many requests you batch. Ask with the model, the quantisation and the expected concurrency and we will say what we have actually seen.
Fine-tuning and training: Both are ordinary work here, and both need considerably more memory than serving the same model: gradients and optimiser state sit alongside the weights. A parameter-efficient method needs a fraction of what a full fine-tune does. Tell us the method and the model and we will size it rather than have you find out after delivery.
Tenancy: One tenant per physical machine and the card passed straight through to your operating system. No hypervisor, no MIG partition, no vGPU profile, no time-slicing scheduler, and nobody else's workload on the card. What you measure on the first afternoon is what you keep.
Bandwidth: Unmetered in both directions at every port speed, with no transfer allowance and no egress line on any invoice. Pulling weights down and serving tokens back out costs nothing beyond the port.
What this page does not price: Cards, or anything else. Every card we fit, what each adds per month and what the machine under it costs are on GPU dedicated servers, each figure against the exact machine it belongs to.