How many GPUs do you actually need? A sizing guide for fine-tuning and inference
Most teams over-buy for training and under-plan for inference. Some simple arithmetic gets you close before you talk to anyone.
GPU sizing looks mysterious, but most of it comes down to memory. If the model, its training state and a batch of data do not fit in GPU memory, nothing else matters. Here are the rules of thumb we use on sizing calls.
Inference: start with the weights
At 16-bit precision, each parameter takes 2 bytes. A 7-billion-parameter model needs about 14 GB just for its weights. A 70B model needs about 140 GB, which means at least two 80 GB GPUs before you add anything else.
Then add the KV cache, the memory that holds conversation context. Long prompts and many simultaneous users can need as much memory again as the weights. Quantising to 8-bit or 4-bit roughly halves or quarters the weight memory, often with little quality loss.
Full fine-tuning: multiply by eight or more
Training keeps the weights, gradients and optimizer state in memory. With the common Adam optimizer and mixed precision, budget around 16 bytes per parameter before activations. A full fine-tune of a 7B model wants well over 100 GB, so plan for a multi-GPU node.
LoRA and QLoRA: the practical option
Parameter-efficient fine-tuning trains small adapter layers and freezes the rest. A 7B or 8B model can be LoRA-tuned on a single 80 GB GPU. With 4-bit QLoRA, even 70B models become workable on one or two GPUs. For most business use cases, this is where we start.
| Job | Model size | Typical starting point |
|---|---|---|
| Inference, 16-bit | 7–8B | 1 GPU (24–80 GB) |
| Inference, 16-bit | 70B | 2–4 × 80 GB GPUs |
| LoRA fine-tune | 7–8B | 1 × 80 GB GPU |
| QLoRA fine-tune | 70B | 1–2 × 80 GB GPUs |
| Full fine-tune | 7–8B | 1 node of 8 GPUs |
| Full fine-tune | 70B | Several 8-GPU nodes with fast interconnect |
Then think about time
Memory tells you the minimum. Time tells you how many more GPUs are worth paying for. Doubling GPUs roughly halves wall-clock time for well-parallelised jobs, until communication overhead catches up. That is why multi-node training needs InfiniBand-class networking.
The cheapest sizing tip
Run one epoch on a small sample on demand, measure throughput, then extrapolate. A day of on-demand time can save you from reserving the wrong cluster for six months. Talk to us about a sizing run.
