LLM VRAM calculator: how many GPUs does your model need?

How it is calculated

W  = P × 10^9 × b_w ÷ 2^30
κ  = 2 × L × H_kv × d_h × b_kv          (bytes per token)
KV = κ × C × U ÷ 2^30

V_total = (W + KV) × (1 + o) + V_fixed
n_raw   = V_total ÷ (V_gpu × u)

round n_raw up to a tensor-parallel degree in {1, 2, 4, 8} that divides kv_heads;
above 8, nodes = ceil(n_raw ÷ 8) with 8 GPUs per node

cloud    = GPUs × hourly rate × 730
on-prem  = GPUs × (price ÷ 36 + power + hosting)
loaded   = GPU cost × (1 + ops overhead)
API cost = tokens (millions) × API price per million
break-even tokens = loaded cost ÷ API price per million

tokens/s per stream ≈ η × total memory bandwidth ÷ (active params × bytes per param)
capacity ≈ streams × per-stream speed × batch efficiency (0.5)

Assumptions

• P is parameters in billions, b_w bytes per weight, L layers, H_kv KV heads, d_h head dimension, b_kv bytes per KV element, C context tokens, U concurrent sessions, o runtime overhead, u usable VRAM share. 1 GiB = 2^30 bytes.
• V_fixed is a fixed runtime allowance from the dataset (2 GiB). The 4-bit bytes per parameter (0.55 and 0.60) are approximate.
• Mixture-of-experts models keep every expert in memory; only speed uses the active parameter count.
• Cloud cost uses the cheapest listed provider for the chosen GPU. No reserved rates are published, so reserved uses on-demand.
• GPU specs and cloud rates were read from NVIDIA and provider pages. NVIDIA publishes no list prices for data-center GPUs, so on-prem purchase prices are third-party or estimated figures; a result that depends on an unverified value carries a DATASET_UNVERIFIED notice naming it.
• Decode speed is memory-bandwidth bound and ignores prefill, interconnect and scheduler effects. The monthly volume is spread evenly over 30 days.

The data behind it

Three ways to reduce this number

1. Store the KV cache in FP8.
2. Cap context length or concurrency.
3. Quantize weights to FP8 or 4-bit and re-measure quality.

Use it from code

The same engine is available as a REST endpoint and as an MCP tool named calc_self_hosted_llm_gpu_vram_estimator. Your scenario as an API call:

curl -X POST 'https://api.x12i.com/v1/calc/self-hosted-llm-gpu-vram-estimator' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
  "input": {
    "model_preset": "custom",
    "quantization": "fp8_int8",
    "kv_cache_dtype": "fp16",
    "context_len_tokens": 16384,
    "concurrent_users": 50,
    "runtime_overhead_pct": 10,
    "usable_vram_pct": 90,
    "gpu_model": "any",
    "deployment": "cloud_on_demand",
    "monthly_tokens_millions": 5000,
    "api_blended_usd_per_mtok": 5,
    "ops_overhead_pct": 20,
    "privacy_sensitive": false
  },
  "options": {
    "explain": false
  }
}'

Copy as API call Schema: GET /v1/calc/self-hosted-llm-gpu-vram-estimator/schema

Frequently asked questions

Related calculators