Your own Ollama server, sized for the model you want

How much memory will your model need?

Enough for the model plus about a quarter again, if one person talks to it at a time. A model's size is its number of parameters multiplied by the space each one takes. Most people run models quantized, which means stored at 4 or 8 bits per parameter instead of 16, shrinking them to roughly a quarter or a half of full size for a small cost in quality. The table uses the files Ollama actually downloads, which run a little heavier than that arithmetic: its standard Llama 3.1 8B, for instance, is stored at 4 bits and weighs 4.9 GB.

That extra quarter assumes a short chat. Ollama holds every token of the conversation in memory while it answers, and it chooses how long that conversation window is from the graphics memory it finds: a card under 24 GiB gets 4k tokens, one from 24 up to 48 GiB gets 32k, and 48 GiB or more gets 256k. On a big card that is far more than a chat needs, and it can push a 70B model partly off the card.

So set the window yourself with OLLAMA_CONTEXT_LENGTH (8192 tokens is plenty for chat, but Ollama's own guide asks for at least 64k for agents and coding tools, and a window that size needs memory of its own) and check the CONTEXT and PROCESSOR columns of ollama ps. 100% GPU means the whole model sits on the card; a reading like 48%/52% CPU/GPU means part of it overflowed into ordinary memory and replies will be slow. Letting Ollama answer four people at once makes the window four times as large, so for a team the LLM sizing page plans half again over the weights, the figure it uses for serving traffic.

Do you need a graphics card?

Not for a small model and a few users. Without a card, Ollama runs the model on the processor from the server's ordinary memory. That is slower than a card, but for a small model and a handful of people it is often enough.

Once the model grows or the users multiply, it belongs in a card's memory. Ollama works with NVIDIA cards from compute capability 5.0 upward (NVIDIA's number for a card's generation) on driver 550 or newer. Older cards between 5.0 and 6.2, such as the Tesla P4, Tesla P40 and Quadro P6000, need driver 570 or newer. NVIDIA's 580 branch is also the last that supports Pascal and Volta cards (the Tesla P4, P40 and V100, the Quadro P6000 and the TITAN V), so for any of them name a 580 driver on the order. AMD Instinct cards on Ollama's list work too, through AMD's ROCm 7 software; of the ones we fit, that means the MI210 and not the older MI50. With 64 GB, the Instinct MI210 is also the smallest single card we fit that takes a 70B model at 4-bit with room for one conversation. The GPU servers page lists every card we fit, with what each one costs.

An AMD card runs ROCm rather than NVIDIA's CUDA. Ollama supports both, but check that the rest of your tools do before you order.

Getting Ollama running, step by step

1. Order the server with Ubuntu, one of 33 system images that install in one click. If it has a card, name the driver version on the order and it arrives installed.
2. Log in and run Ollama's install command. It sets Ollama up as a background service that starts with the server and restarts itself if it ever stops.
3. Download a model and ask it a first question.
4. Check where the model landed with ollama ps. 100% GPU means the whole model sits in the card's memory, which is where you want it.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b "In one sentence, what can you help me with?"
ollama ps

If something refuses to start, the service log usually says why: run journalctl -e -u ollama and read the last lines.

Keeping strangers away from your model

Out of the box, Ollama listens only to the server itself, on address 127.0.0.1 and port 11434, so nobody on the internet can reach it. That default is worth keeping. Ollama has no password of its own: its OpenAI-style API asks the app for a key and then ignores whatever key it is given, so an open port lets anyone use your model.

To use it from your own laptop, forward the port over SSH. Your laptop then finds Ollama at http://localhost:11434, and chat apps or code written for OpenAI connect to http://localhost:11434/v1/; the key they ask for can be any word you like.

ssh -N -L 11434:127.0.0.1:11434 you@your-server

If your other servers need to call it, switch the firewall on for those servers and nobody else first, and only then tell Ollama to listen on every address. The firewall lines are Ubuntu's ufw; swap the example address for your own. The three settings in the override open Ollama to the network, keep the model loaded and keep the conversation window short.

One thing a firewall cannot do is hide what is said. Those calls travel across the internet as plain HTTP, readable along the way, so for anything private run them through an SSH tunnel or WireGuard between the servers, or put a reverse proxy with HTTPS and a login in front of Ollama.

sudo ufw allow 22/tcp
sudo ufw allow from 203.0.113.10 to any port 11434
sudo ufw enable

sudo systemctl edit ollama.service
# at the top of the file that opens, add:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Let SSH through the firewall before you switch it on. If you do lock yourself out, the browser console of the server's management controller still gets you in without going through its network. Every rack server and blade has one; add a console login in the dashboard to open it. Most Moonshot cartridges have power control only, so on those, check the SSH rule twice.

Keeping it quick every day

• No more waiting for a reload. Ollama drops a model from memory 5 minutes after it was last used, and the next question waits while it loads again. Setting OLLAMA_KEEP_ALIVE=-1 holds it in memory for good, while a value such as 24h holds it for a day.
• More than one person at a time. By default Ollama answers one request at a time per model and queues the rest. OLLAMA_NUM_PARALLEL lets it answer several together, and each extra one costs another conversation window of memory.
• The version you tested stays put. Running the install command again upgrades to the newest Ollama; run it with OLLAMA_VERSION set to a release number to stay on that release. Nothing on our side changes it for you.
• Models where you want them. Downloads are saved to /usr/share/ollama/.ollama/models. To keep new ones on a bigger disk, create a folder there, hand it to the service's user with sudo chown -R ollama:ollama /path/to/folder, and point OLLAMA_MODELS at it in the same override. Anything you downloaded before stays in the old folder; copy it over or download it again.

Why run it on a whole machine?

Because a model server is always on, and it runs best with nobody else on the hardware. Every dedicated server here is one physical machine with one customer: no virtual slices, nothing oversubscribed, and the speed you measure on day one is the speed you keep.

Your prompts and documents stay on hardware only you can log into. Choose Amsterdam or Bucharest to keep the data in the EU, or New York, Miami or San Francisco to keep it in the US.

The port is unmetered, so downloading a large model twice or answering a busy team never turns into an overage charge. One month is the default term with no contract, and memory, disks or a faster port can be added to the machine you already have, with no move and no new IP address.

Help is there around the clock, by telephone and chat. On chat, a reply usually comes in under a minute: from a person when one is on shift, and otherwise from an assistant that can read your account and your server's hardware health. Asking for a person always gets one.

When a server is not the right answer

If you only need a model now and then, or you need the most capable model on the market, paying per use through an API costs less and asks nothing of you. A server of your own pays off in three cases: your usage is regular, your data must not leave your hands, or a freely available model is already good enough.

Some models also outgrow one card. A 70B model at 8-bit, with room to serve, is too big for one card on our list, which makes it a multi-card build that we quote on request rather than sell from a checkout page.

Questions people ask

Do I need a GPU to run Ollama on a server?

Not for small models. Ollama falls back to the processor and ordinary memory when there is no card, which suits one person or a few people using a small model. For bigger models, or many people at once, the model needs to sit in a card's memory.

What does a 70B model need to run well under Ollama?

Llama 3.1 70B, as Ollama fetches it at 4 bits per parameter, takes 43 GB. With room for one person's conversation, plan on about 54 GB, more than a 48 GB card holds, and set the conversation window short: 8192 tokens is plenty for chat. The smallest single card we fit that holds it is the 64 GB AMD Instinct MI210. At 8-bit the download is 75 GB, and with room to serve that becomes a multi-card build.

Can people outside my office use my Ollama server safely?

Only with protection in front of it, since Ollama itself never asks who is on the other end. Keep it listening on the server itself and reach it through an SSH tunnel, let only your own servers' addresses past the firewall and encrypt that traffic, or add a login and HTTPS in front of it with a reverse proxy.

Will apps written for the OpenAI API work with it?

Most will. Ollama answers the same kind of requests as OpenAI's API for chat, completions, embeddings and model lists. With an SSH tunnel running, the app's address is http://localhost:11434/v1/ and any key will do; if a reverse proxy sits in front, give the app that proxy's address and the login it expects.

How soon can my Ollama server be ready?

About 30 minutes after your order is approved if a machine already in the rack matches it, with the card you want or with none. Otherwise it is built: 4 to 24 hours when we hold the card and parts, 5 to 10 working days when they have to be bought in, and 10 to 15 for a scarce or current-generation card on allocation. Installing Ollama then takes one command.

Am I tied into a contract?

No. One month is the default term, and you can add memory, disks or a faster port to the same machine later without moving. If we have to buy a card or other part for your order, we ask for three months up front. A refund is possible on written request within three calendar days of first activation, once troubleshooting is exhausted, pro rata less 15%.