If you’ve ever tried to run an open-source model on your own machine, you’ve hit this question fast: will it actually fit? Downloading 40GB of weights only to watch your GPU throw an out-of-memory error is a frustrating way to find out the answer is no.
To make that guesswork disappear, I built a free LLM Hardware Calculator. You pick a model, a quantization level, your context length, and how many requests you want to run in parallel — and it tells you how much VRAM and system memory you actually need. No spreadsheets, no napkin math.
This post explains how local-LLM memory requirements work, walks through some real examples, and shows you how to size your hardware before you spend a cent on a GPU.
Why “will it run?” is harder than it looks
The size of a model on Hugging Face is not the amount of memory it needs to run. A 7-billion-parameter model isn’t “7GB.” The real requirement depends on four things:
- Parameter count — how big the model is (7B, 70B, 405B, and so on).
- Quantization — how many bits each weight is stored in.
- Context length — how much text the model holds in memory at once.
- Batch size — how many requests you’re serving in parallel.
Get any one of these wrong and your estimate can be off by 2–4×. That’s the gap between “runs comfortably on a gaming GPU” and “needs a data-center card.” The calculator handles all four for you, but it’s worth understanding what’s happening underneath.
The three things that eat your VRAM
Total memory use comes down to three components.
1. Model weights
This is the big one. The formula is simple:
Weights (GB) ≈ parameters × bits per weight ÷ 8
At full 16-bit precision (FP16/BF16), every parameter takes 2 bytes. So:
- 7B model: ~14 GB
- 13B model: ~26 GB
- 70B model: ~140 GB
- 405B model: ~810 GB
That’s why nobody runs large models at full precision on consumer hardware — which brings us to quantization.
2. KV cache (the part people forget)
As the model generates text, it stores a “key-value cache” for everything in the context window. This cache grows linearly with context length and batch size. A model that fits neatly at 4K context can overflow your GPU at 128K context, even though the weights never changed. If you’re planning to use long prompts or serve multiple users, this is often the deciding factor.
3. Activation overhead
There’s a smaller amount of working memory needed during inference — roughly 10% of the model size. Small, but enough to tip you over the edge if you’ve cut it close.
The LLM Hardware Calculator sums all three so you see the true total, not just the weight file size.
Quantization: how to fit big models on small GPUs
Quantization reduces the precision of each weight — for example, from 16-bit floats down to 4-bit integers. This shrinks the memory footprint dramatically:
| Precision | Bytes/param | 7B model | 70B model |
|---|---|---|---|
| FP16 (16-bit) | 2 | ~14 GB | ~140 GB |
| Q8 / INT8 (8-bit) | 1 | ~7 GB | ~70 GB |
| Q4 (4-bit) | ~0.5 | ~3.5 GB | ~35 GB |
Q4 is the sweet spot for most people. It cuts memory roughly 4× versus FP16 with only a modest quality drop, which is why it’s the most common choice for running large models on consumer GPUs. Go lower (Q3, Q2) and you save more memory but start to see noticeable quality loss on harder tasks.
The practical takeaway: quantization is what makes a 70B model runnable on a single 24GB card that could never hold it at full precision.
Real-world examples
Here’s roughly what you’re looking at for a few popular models (inference only, modest context). Plug your exact settings into the calculator for precise numbers.
Llama 3.1 8B
- Q4: ~5–6 GB → runs on almost any modern GPU, even an 8GB card
- FP16: ~16 GB → needs a 16–24GB card
Llama 3.3 70B
- Q4: ~40 GB → needs a 48GB card or two 24GB GPUs
- Q8: ~70 GB → data-center territory
Qwen 2.5 Coder 32B
- Q4: ~18–20 GB → fits comfortably on a 24GB GPU
DeepSeek / large MoE models (100B+)
- Even quantized, these need serious VRAM or multi-GPU setups — exactly the kind of thing you want to check before downloading hundreds of gigabytes.
What GPU should you actually buy?
A rough map from VRAM to hardware:
- 8–12 GB (RTX 3060, 4060 Ti) — small models up to ~8B at Q4. Great for getting started.
- 16 GB (RTX 4060 Ti 16GB, 4070 Ti Super) — comfortable with 8–13B models, tight on larger ones.
- 24 GB (RTX 3090, 4090) — the consumer sweet spot; runs 32B models well and 70B at aggressive quantization.
- 32 GB+ (RTX 5090) — headroom for larger models and longer context.
- Apple Silicon — unified memory means a 64GB or 128GB Mac can load big models, though generation speed is bound by memory bandwidth.
- 48–80 GB (A6000, A100, H100) — for 70B+ at higher precision or serving many users.
Before buying, run your target model through the LLM Hardware Calculator at the context length and batch size you actually plan to use. It’s a lot cheaper than returning a graphics card.
One important caveat: inference vs. training
Everything above assumes inference — just running the model. If you want to fine-tune or train, multiply your VRAM estimate by roughly 4–8× to account for gradients, optimizer state, and activations. A model that runs happily on one GPU for inference may need a whole cluster to train.
Frequently asked questions
How much VRAM do I need to run a 7B model? Around 4–6 GB at Q4 quantization, or ~14 GB at full FP16 precision. Most modern GPUs can handle a 7B model at Q4.
Can I run a 70B model on a single 24GB GPU? Yes, but only with aggressive quantization (Q4 or lower) and modest context, and it’ll be tight. A 48GB card or a dual-GPU setup is much more comfortable.
Does context length really affect memory that much? It can. The KV cache grows linearly with context length and batch size, so jumping from 4K to 128K context — or serving many parallel requests — can add gigabytes on top of the model weights.
What quantization level should I use? Q4 is the default recommendation for most people: big memory savings, minimal quality loss. Use Q8 or FP16 if you have the VRAM and want maximum quality; drop to Q3/Q2 only when you’re desperate for space.
Is this calculator for inference or training? The estimates are for inference. For training, expect to need several times more memory.
Try it yourself
Sizing hardware for local LLMs doesn’t have to be guesswork. Pick your model, set your quantization and context, and get a straight answer in seconds:
👉 Open the free LLM Hardware Calculator
Whether you’re deciding which GPU to buy, checking if a model fits your current setup, or planning a multi-user deployment, it’ll save you the trial-and-error. If you find it useful, share it with someone who’s about to buy a GPU they might not need.
