Your own self-hosted Whisper server for private transcription

Which Whisper model will fit?

Most of them, as long as you choose the right software. Whisper comes in sizes from tiny to large, and the bigger ones are more accurate but need more graphics memory (VRAM). OpenAI's README lists approximate needs of about 5 GB for medium and about 10 GB for large, though a later test measured OpenAI's own code at about 4.7 GB for large-v2.

faster-whisper is a reimplementation of the same models that runs up to four times faster and uses less memory, above all at int8. Its maintainers ran a 13-minute recording through the large-v2 model on an 8 GB graphics card and recorded:

int8 is a compressed form of the model weights. It cut memory by about a third and, in that test, was no slower. The last row needs more than 6 GB, so on a 6 GB card use int8 when batching. Times on your own card will be different, so try a real recording before you settle on a model.

Do you really need a graphics card?

Not for small jobs. faster-whisper also runs on an ordinary processor, and the small model at int8 got through the same recording in 1m42s using eight threads of a desktop chip. Once you are transcribing hours a day, or want the large model's accuracy, the graphics card is what keeps the queue moving.

One thing to know about the turbo model: it is quick, but OpenAI says it was not trained to translate. If you need foreign-language speech turned into English text, stick with large or medium.

The server that fits

Our RTX 3050 GPU server is built for jobs like this: a whole desktop computer, its graphics card an RTX 3050 with 6 GB, plus a six-core Intel Core i5-9500, 32 GB of memory and a 256 GB NVMe drive, racked in Bucharest on an unmetered gigabit port. You rent it whole. The card goes straight to your operating system, so all 6 GB are yours and nobody else's job slows yours down.

Because nothing counts the traffic, uploading long recordings never adds to the bill, and the price is the same whether the card is busy all day or idle. Once your order is approved, the machine is usually online about 30 minutes later, with root access and Linux or Windows.

Thinking of a chat model on the same box too? Our Ollama guide covers that, but a 4-bit 8B model takes nearly 5 GB on its own, so it will not share 6 GB with the large Whisper model; pair it with a small Whisper model, or run one at a time. Need more memory than that? Our GPU servers page lists bigger cards.

Up and running in an evening

1. Order the server with Ubuntu, install NVIDIA's driver, and check that nvidia-smi lists the RTX 3050.
2. Make a Python virtual environment and run pip install faster-whisper.
3. Add NVIDIA's cuBLAS and cuDNN 9 libraries, which faster-whisper needs on a graphics card. Its README installs them with pip and sets LD_LIBRARY_PATH before Python starts.
4. Load a model once and keep it loaded, then send it files. The short script below transcribes one recording and prints each line with its timestamps.

from faster_whisper import WhisperModel

# load once; reuse for every file
model = WhisperModel("large-v3", device="cuda", compute_type="int8")

segments, info = model.transcribe("interview.wav", vad_filter=True)
for seg in segments:
    print(f"{seg.start:7.1f}  {seg.text}")

Setting vad_filter=True uses the Silero VAD model to cut out silences longer than two seconds (tunable with vad_parameters) before Whisper sees them, which saves time on interviews and meetings full of pauses. If you prefer containers, the README notes that NVIDIA's official CUDA 12 runtime image with cuDNN 9 already contains cuBLAS and cuDNN, and names the exact tag.

Keeping it private and safe

Want your existing apps to call it the way they call OpenAI's transcription API? Put an OpenAI-compatible server in front. The faster-whisper README lists speaches, which runs faster-whisper behind such an endpoint, deploys with Docker and works with OpenAI's SDKs. The point of hosting Whisper yourself is that the audio stays with you, so do not undo that by leaving the door open. Once others can send files, keep a few habits:

• Put it behind HTTPS and ask for a key or a login. An open endpoint is free transcription for strangers.
• Cap the upload size and handle files one after another, so one three-hour file does not block everyone.
• Delete the audio once the text is saved, unless you truly need it later.
• The server has a single drive, so copy finished transcripts elsewhere, or add the nightly backup on the order form.

These desktops have no remote console. You can power them on remotely, and if the system freezes, our team in Bucharest restarts it for you.

When something else suits you better

If you only transcribe a file now and then, run the small model on your own laptop for free. If you want to train or fine-tune models, 6 GB is too little; this machine is meant for running them. And if your users are far from Europe, remember every one of these desktops is in Bucharest; our data centres page shows where everything else we run is.

Questions people ask

Will the large Whisper model run on a 6 GB card?

Yes, with faster-whisper. In its maintainers' test, large-v2 used about 2.9 GB at int8 and 4.5 GB at fp16. OpenAI's README lists about 10 GB for large as a rough figure, though the same test measured OpenAI's code at about 4.7 GB, so faster-whisper at int8 is the comfortable choice on 6 GB.

How fast will my transcripts come back?

It depends on the card, the model and your audio. In faster-whisper's own test, an RTX 3070 Ti with 8 GB finished a 13-minute recording in about a minute, or 16 seconds with batching. Time a few of your own files to know for sure.

Can I use Whisper in a paid product?

Yes. OpenAI publishes Whisper's code and models under the MIT licence, which allows commercial use, and faster-whisper uses the same licence.

Do my recordings leave the server?

Not if you run Whisper yourself. The model works entirely on your machine, so nothing is sent to an outside service unless your own code sends it.

Can it translate other languages into English?

The multilingual models can. The fast turbo model was not trained for translation, so pick large or medium for that.

Is int8 less accurate than fp16?

It can change the odd word, because the weights are compressed. Run the same recording both ways and keep int8 if you are happy with the text, since it uses about a third less memory.