An LLM with seven billion parameters does not have one fixed
memory requirement. Precision changes the weight footprint. Context length and
concurrent sequences change the KV cache. The serving engine adds workspaces
and allocator overhead. A model that loads successfully can still run out of
memory under production traffic.
The safest way to size GPU VRAM for LLM inference is to
estimate each memory category, preserve headroom, and confirm the result with
the actual runtime.
Begin with model weights
A simple uncompressed estimate is:
parameter count x bytes per parameter.
At FP32, each parameter is roughly four bytes; FP16 or BF16
is roughly two. An uncompressed 7B model is therefore about 14 GB at 16-bit
precision before runtime overhead. A 70B model is about 140 GB on the same
rough basis.
Quantization lowers weight memory. Eight-bit weights
approach one byte per parameter and four-bit weights approach half a byte, but
real files and loaded tensors include scales, zero points, group metadata,
padding, and sometimes higher-precision layers. The advertised bit width is not
a precise VRAM measurement.
Architecture also matters. Mixture-of-experts models may
contain many total parameters while activating a subset for each token, yet the
serving system may still need to store a much larger weight set. Read the model
and engine documentation rather than applying a dense-model shortcut blindly.
Add the KV cache
Transformer inference stores attention keys and values for
tokens already processed. This KV cache enables the model to generate the next
token without recomputing the full history. Its size grows with the number of
layers, KV heads, head dimension, bytes per cache element, total cached tokens,
and concurrent sequences.
A conceptual estimate is:
layers x 2 (key and value) x KV heads x head dimension x
bytes x cached tokens.
Grouped-query or multi-query attention reduces KV heads
relative to query heads, so two models with similar parameter counts can have
very different cache requirements. Some engines support lower-precision KV
caches, but support and quality should be validated.
"Maximum context" is not the same as typical
context. Capacity planning needs a distribution: prompt lengths, generated
lengths, concurrency, and admission policy. One 32,000-token conversation and
thirty-two 1,000-token requests may contain a similar number of cached tokens
but create different scheduling and latency behavior.
Reserve runtime headroom
After weights and KV cache, leave room for CUDA context,
kernels, attention workspaces, temporary activations, graph capture, logits,
sampling, communication buffers, and memory fragmentation. Serving engines
often reserve a configurable fraction of GPU memory for cache blocks while
protecting enough space for execution.
There is no universal overhead percentage. It changes with
engine, model architecture, attention backend, batch shape, CUDA graphs,
speculative decoding, and GPU. Treat a percentage as a planning estimate, never
as a measured fact.
Test the worst allowed request, then run sustained
concurrent load. Watch both allocated and reserved memory, because framework
allocators can hold blocks after individual tensors are released.
A transparent sizing example
Consider a hypothetical 13B dense model served with nominal
4-bit weights. The ideal arithmetic for weights is about 6.5 GB. Metadata,
non-quantized components, and loader behavior raise the actual footprint, so
use the model artifact and a test load to establish the real number.
Next calculate KV cache from the architecture configuration
rather than guessing from parameter count. Multiply the per-token cache by the
maximum total active tokens the scheduler will admit. Add measured runtime
overhead and a safety margin. If the total approaches a GPU's physical
capacity, reduce concurrency or context, choose a more efficient cache format,
or move to a larger-memory GPU.
This is an estimate, not a Hostnot GPU performance claim.
The same method applies to any provider because it is based on model state and
runtime behavior.
Quantization is a system decision
Quantization can turn an impossible deployment into a
practical one, but memory saving is only part of the trade. Compare model
quality on representative tasks, kernel availability, load time, prefill and
decode performance, and operational support.
Formats are not interchangeable. A weight-only format
optimized for one engine may be slow or unsupported in another. Some methods
require calibration; others are produced during or after training. CPU offload
may make a model load, but frequent transfers can cause latency that is
unacceptable for an interactive service.
Keep the unquantized or reference evaluation results.
Without them, a lower-memory deployment can silently save money by delivering
worse answers.
Single-GPU versus multi-GPU memory
If the model does not fit one GPU, tensor parallelism can
shard portions across devices. Pipeline parallelism assigns layers to stages.
Some systems combine these with data parallel replicas. Each strategy has
communication and scheduling costs, and some state remains replicated.
Do not add GPU capacities and assume they form one
transparent pool. Confirm that the serving engine supports the selected
parallel size and model architecture. Check whether GPUs share a fast
interconnect or communicate only through PCIe, and test latency at realistic
prompt lengths.
Sometimes one 80 GB or larger accelerator is operationally
simpler than several smaller cards. Sometimes several lower-cost cards deliver
better throughput. Only a complete-machine benchmark can distinguish them.
Turn memory into a cloud shortlist
Once the estimate is stable, create tiers. A 24 GB GPU may
be suitable for a smaller quantized model or modest concurrency. The 32 to 48
GB range gives more cache and model headroom. A100, H100, H200, B200, and other
data-center classes provide larger per-GPU memory options for heavy inference.
Hostnot GPU publishes per-GPU VRAM alongside synchronized
configurations at https://hostnotgpu.ae/gpus.
Its directory also warns that GPU count is not a pooled-memory promise and asks
buyers to examine the complete machine. Availability and listed rates can
change, so use the catalog as a current shortlist, then confirm the exact
offer.
Validate under production-shaped load
Load the exact revision and tokenizer, pin all serving
settings, and define maximum prompt and output tokens. Generate a request set
that resembles real traffic, including long-tail inputs rather than only short
prompts.
Increase concurrency gradually while recording:
- peak
VRAM and cache utilization;
- accepted
and queued requests;
- time
to first token and inter-token latency percentiles;
- throughput
and failed requests;
- quality
changes from precision or quantization;
- cost
per successful output at the target service level.
Test admission controls. A service should reject, truncate,
or queue requests predictably before memory exhaustion destabilizes every
active request. Also test model reloads and rolling updates, which may
temporarily require additional memory or capacity.
Conclusion
GPU VRAM for LLM inference equals more than weight size.
Model representation establishes the baseline; KV cache adds a
traffic-dependent term; the runtime requires workspaces and safety headroom.
Quantization, context, concurrency, and parallelism change the answer.
Use formulas to rule out impossible choices, then measure
the real model under production-shaped load. That process yields a defensible
GPU size and an explicit admission policy instead of a fragile deployment that
merely succeeds at startup.