GPU & AI Server Sizing

What it works out

GPU sizing is a memory problem before it is a compute problem. A model that does not fit does not run slowly, it does not run — and the memory a deployment needs is rarely what the parameter count suggests, because every concurrent request carries its own cache.

• Model weights, from parameter count and precision
• KV cache, from context length, concurrent requests and the model's depth
• Allocator and activation overhead, and the headroom you choose to keep free
• The smallest RTX PRO configuration that carries the total
• System RAM and NVMe to match, and the network a single node wants

Why it shows the arithmetic

Any calculator can print a card count. This one prints the derivation, because the interesting result is usually which term dominates: on a busy deployment the cache commonly exceeds the weights, and that changes what you would do about it. Every coefficient is visible and every assumption is stated.

• Weights, cache, overhead and headroom shown separately, with the figures used
• The effective context and concurrency each workload sizes against
• A warning when the cache, not the model, is setting the number
• Honest limits: past four cards, or training from scratch, the answer is a conversation

What it does not do

It sizes a single node on PCIe, which is the machine most teams actually want and the one that can be quoted without a network design. It does not size multi-node training clusters, it does not propose data-centre Blackwell parts, and it does not publish prices.

The model is rarely the thing that fills the card

Most people size a GPU server by looking up the model. A 70B model at FP16 needs about 130 GB for its weights, so two cards, and that feels like the answer. It usually is not, because weights are the part that does not change. Every person talking to the model at once adds memory that the weights figure never mentions.

That second number is the KV cache, and on a busy deployment it routinely exceeds the model itself. Sizing without it is how a server arrives that runs beautifully in testing and falls over on the day it is given to a department.

What the KV cache is, in plain terms

A language model re-reads its entire conversation to produce each new token. Doing that literally would be hopelessly slow, so it keeps the intermediate result for every token it has already seen — two tensors per layer, called the key and value cache. That cache lives in GPU memory, it grows with every token in the window, and there is one of it per request in flight.

So the cache scales with three things at once: how long the context is, how many people are using it, and how deep the model is. Double the context and it doubles. Double the users and it doubles again. This is why a small model serving a large team can need more memory than a large model serving one person.

• Long context is the expensive axis — a coding assistant with a repository in the window costs more than another ten chat users
• Modern models use grouped-query attention, which cuts the cache to a fraction of the textbook figure. Sizing that ignores it overstates memory several times over
• Quantising weights to 4 bits does not quantise the cache — most serving stacks keep it at 8 or 16 bits

Why the use case changes the answer

The same model and the same user count give different numbers depending on what the deployment is for, and the difference is not marketing. A chat assistant holds a short window briefly. A reasoning model generates for a minute and holds its cache the whole time. A RAG system puts retrieved passages in front of every question, so the effective context is larger than anything the user typed. An agent runs several steps, each carrying accumulated tool output.

The calculator applies a bias for each of these and shows you the number it used, because a multiplier you cannot see is a number you cannot check.

Precision, and what you actually give up

Precision is the cheapest lever available. FP16 stores two bytes per parameter, FP8 one, INT4 half of one — so the same model can need a quarter of the memory depending only on how it is stored. For most inference workloads the quality difference between FP16 and FP8 is not measurable in the output, and the memory difference decides whether a build needs one card or three.

Fine-tuning is the opposite case. Training holds the weights, a gradient for each of them, and the optimiser's state — roughly four copies in memory — which is why a model that serves comfortably on one card may need several to be trained on.

Fitting is not the same as keeping up

Memory answers whether the model fits. It says nothing about whether the machine serves the request rate, and those are different questions with different answers. A small model at high concurrency fits comfortably on one card and can still need three to keep up — the weights were never the constraint, the token rate was.

So the sizing runs on both axes and recommends the larger. Where throughput is the binding one, the page says so, because it changes what you would do about it: a memory-bound build wants a bigger card, a compute-bound build wants more of them, and occasionally a cheaper board bought in quantity is the better answer.

• Memory-bound: the model barely fits — a larger card, or lower precision, is the lever
• Compute-bound: it fits easily and cannot keep up — more cards, and the consumer board becomes worth discussing
• Fine-tuning is neither: it runs to completion rather than serving a rate, so it is sized on memory alone

Power, and the two numbers people confuse

GPU servers are a facilities problem as much as a technical one. Four high-end cards draw more than most office circuits deliver, and a build that cannot be powered where it needs to live is not a build. The sizing therefore returns typical draw, peak, the power supply to specify, and the heat the room has to remove.

Two of those are routinely conflated. The power supply is rated on the DC load it must deliver, so it is sized on peak draw plus a margin. What the circuit and the UPS see is that load divided by supply efficiency, which is a larger number. Sizing a UPS from the PSU label, or a PSU from the wall figure, both go wrong in the direction that matters.

Headroom is a decision, not a constant

A card filled to its last gigabyte will run until something changes: a longer prompt, one more user, a serving stack update that allocates differently. The calculator reserves twenty percent by default and gives you the control, because the right figure depends on how predictable your load is. A fixed internal tool can run tighter than a service exposed to a whole organisation.