GPU requirements for fine-tuning your own LLM, and the card that fits it

What fine-tuning actually asks of the card

You are not rewriting the model's billions of numbers. With LoRA you freeze the original and train small extra layers, called adapters, that learn the change you want. Hugging Face PEFT describes it as training low-rank update matrices while the original weights stay untouched, which is why the result is small and portable.

QLoRA is the same idea with one twist: the frozen original is loaded in 4-bit through the bitsandbytes library. By Unsloth's account that cuts the memory to about a quarter of 16-bit LoRA, at the price of runs that go a bit slower and lose a little accuracy. Unsloth suggests you start there, with a small instruct model such as Llama 3.1 8B.

Training on a 4-bit model means training adapters only. Hugging Face's Transformers docs are explicit that 8-bit and 4-bit training works only for extra parameters, so a full fine-tune needs the model loaded unquantized.

How much card memory will your model need?

Find your model size in the left column. The two middle columns are Unsloth's published figures, which Unsloth labels the absolute minimum, adding that certain models take more. The right column is our own reading of which card memory clears each one.

Two more published figures point the same way. According to the PEFT docs, one 48 GB GPU is enough for QLoRA on a 65B model, and Unsloth's own guide says a 70B fits in under 48 GB with QLoRA.

Why a run that fits can still crash

The minimums assume modest settings. A run that loads fine and then stops with an out-of-memory error on the first step almost always has one of these turned up too far. Try the fixes in this order:

1. Lower the per-device batch size. Unsloth names it the main driver of memory use and suggests 1, 2 or 3 when you hit the wall.
2. Raise gradient accumulation to win back the batch you gave up. It gives the same effective batch over more steps instead of more memory.
3. Shorten the context. Unsloth tests at 2048 tokens even on models that accept far longer.
4. Turn on gradient checkpointing. Unsloth's own mode, use_gradient_checkpointing = "unsloth", saves about 30% more by its figure.
5. Drop the LoRA rank. Higher ranks cost memory, and for a fast run Unsloth's advice is 8 or 16.

For a sense of how far careful settings stretch a card: the Transformers documentation fine-tunes Llama-13b on a 16 GB T4 with nested 4-bit quantization, 1,024-token sequences, a batch of 1 and 4 accumulation steps.

Will your card run the software?

Memory is not the only gate. On NVIDIA, Unsloth needs a card of compute capability 7.0 or newer, the generation of the V100, T4 and RTX 20 series onwards, and says the older GTX 1070 and 1080 run only slowly. The bitsandbytes library behind QLoRA reaches back to compute capability 6.0 for its 4-bit formats.

In plain terms: most cards on our list from the Tesla T4 and Quadro RTX parts upward are fair game, and the Pascal-era Tesla P4, P40 and P100 are not where you want to start with Unsloth. Unsloth and bitsandbytes also document AMD support through ROCm; read their AMD guides against the exact card before you choose an Instinct part.

The server that fits

A training run wants the same things for hours on end, and that is what a dedicated GPU server gives you:

• The whole card, nobody else on it. Your own operating system drives it directly, with no partition and no scheduler deciding when your turn is. The memory figure on the card is the memory you get.
• Your versions stay put. You install and pin your own driver and CUDA release, so an update cannot land in the middle of a run.
• No meter on the port. Pull base models and datasets in and copy checkpoints out without counting gigabytes.
• A way back in. The server's HPE iLO or Dell iDRAC gives you the screen and power even if a driver install takes the network down.
• No contract. One month is the default term.

Pick from the accelerator shelf when a run has to finish: those cards carry ECC memory, and the /gpu page names that shelf for fine-tuning. GeForce cards lack ECC, which is fine for experiments where a wrong bit only means starting again.

How soon you can start depends on the card: about 30 minutes if a racked machine already carries it, 4-24 hours when we fit a card we hold, 5-10 working days when one has to be bought in, and 10-15 for a current-generation accelerator on allocation.

Your first training run, step by step

1. Install Linux and the NVIDIA driver, then check that nvidia-smi lists your card.
2. Create a Python virtual environment (on Ubuntu, install the python3-venv package first) and install TRL with PEFT support plus bitsandbytes for 4-bit loading, then run trl env to confirm it sees the card and the CUDA version (first block below).
3. Start a small QLoRA job with TRL's command line. This one uses the settings from TRL's own QLoRA example on a tiny model, so a mistake costs minutes, not hours (second block below).
4. Watch the training loss as it runs; when it finishes, the adapter is in the output folder.
5. Serve the result: load the adapter with its base model in a vLLM server, or convert it for a llama.cpp server.

python3 -m venv ~/finetune
source ~/finetune/bin/activate
pip install "trl[peft]" bitsandbytes
trl env

trl sft \
  --model_name_or_path Qwen/Qwen2-0.5B \
  --dataset_name trl-lib/Capybara \
  --load_in_4bit \
  --use_peft \
  --lora_r 32 \
  --lora_alpha 16 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 16 \
  --output_dir Qwen2-0.5B-QLoRA

Once that works, swap in your own model and dataset. If memory is tight, Unsloth is worth the switch: its README claims 2x faster training with 70% less VRAM, and it has a browser interface, Unsloth Studio.

If you use Unsloth Studio, keep it private. Start it on the server's loopback address with unsloth studio -H 127.0.0.1 -p 8888 and open it from your computer through an SSH tunnel such as ssh -L 8888:127.0.0.1:8888 you@your-server. Unsloth warns that its server-side tools are on by default.

Growing past one card

Adding a second GPU usually makes training faster rather than letting a bigger model fit. In Distributed Data Parallel, the common mode, each GPU keeps a complete copy of the model and works on different samples. TRL's command line takes --num_processes to spread a job across cards, and Unsloth starts the same kind of run with torchrun.

To fit a model that is too large for any one card, it has to be split across them; Unsloth does this with device_map = "balanced" and says its multi-GPU setup still takes manual work. The largest single card we fit holds 96 GB. Beyond that you need more than one card in one machine, which we quote rather than configure: servers with more than one GPU explains the options.

Questions people ask

Which card should I start with for an 8B model?

For QLoRA, a 16 GB card clears Unsloth's 6 GB minimum with room for a real batch. For 16-bit LoRA the minimum is 22 GB; a 24 GB card works but leaves little spare. Those card sizes are our own inference from Unsloth's table, not figures it publishes per card.

Can I fine-tune a 70B model on a single card?

Yes with QLoRA, on a 48 GB card or larger: Unsloth's minimum is 41 GB. With 16-bit LoRA it lists 164 GB, beyond the biggest card we offer, so that run needs a multi-card build.

How long does a fine-tuning run take?

It depends on the model, the dataset size, the card and your settings, so we publish no figure. Unsloth's guide uses a 60-step run to test a setup and 1 to 3 epochs for a real one. Time a short run on your own data and scale from there.

Does training in 4-bit hurt quality?

A little. Unsloth calls QLoRA marginally less accurate than 16-bit LoRA and says its dynamic 4-bit models largely close the gap. It also advises training in the precision you plan to serve in.

What do I keep at the end?

An adapter: TRL's docs describe it as only a few MB, where the full model runs to several GB. You load it on top of the base model, or merge the two with PEFT's merge_and_unload() to get one standalone model.