What is quantization in machine learning?
Quantization stores a model's numbers in fewer bits, for example 8-bit or 4-bit integers instead of 16-bit floats. Each block of weights keeps a scale, and a weight is saved as a small integer that is multiplied by that scale when the model runs. The model gets 2 to 4 times smaller than fp16 at the cost of a small, controlled rounding error.
Quantization is how large language models run on a single GPU, a laptop, or a phone. Serving systems load 4-bit or 8-bit versions of open models to fit more requests per card, and local tools such as llama.cpp ship nearly every model in quantized form. Because decoding one token at a time is limited by memory bandwidth, smaller weights also make generation faster. QLoRA also fine-tunes models on top of a frozen 4-bit base.
Key ideas
- Weight memory is parameters × bits ÷ 8, so going from fp16 to int4 cuts it by 4×.
- Floats trade range for precision through the exponent and mantissa split; bf16 keeps fp32's range with less precision.
- Absmax quantization maps the largest weight to the top integer level and rounds the rest to a uniform grid.
- One outlier stretches the scale for every weight that shares it, so per-group scales protect the rest at a small storage cost.
- Weight-only quantization speeds up memory-bound decoding, while W8A8 also speeds up compute-bound work.
- Methods such as GPTQ, AWQ and NF4 keep 4-bit models close to full precision where plain rounding falls short.
Common questions
What is the difference between post-training quantization and quantization-aware training?
Post-training quantization (PTQ) converts a finished model, sometimes with a few hundred calibration samples, and takes minutes to hours. Quantization-aware training (QAT) simulates the rounding during training so the weights adapt to it. QAT usually holds up better at very low bit widths, but it needs the training pipeline and much more compute, so most LLM releases use PTQ.
How much accuracy does 4-bit quantization lose?
With a good method such as GPTQ or AWQ, 4-bit weight-only models usually lose a small amount of perplexity and score close to fp16 on most benchmarks. 8-bit weights are close to lossless. Below 4 bits, quality drops faster, and smaller models suffer more than large ones because each weight carries more of the model's knowledge.
What do GGUF names like Q4_K_M mean?
GGUF is the model file format used by llama.cpp. Q4 means about 4 bits per weight, K marks the k-quant family, which stores small blocks of weights with their own scales inside larger super-blocks, and M is a medium mix that keeps some sensitive tensors at higher precision. The actual average is a little above the nominal bit count.
Can a quantized model be fine-tuned?
Not directly, because rounding to a few levels destroys small gradient updates. QLoRA solves this by freezing the 4-bit NF4 base and training LoRA adapters in 16-bit precision on top. The base is dequantized on the fly in each forward and backward pass, which lets a 65B model fine-tune on one 48 GB GPU.
Related explainers
- LoRA & Parameter-Efficient Fine-Tuning — Adapt large models by training tiny low-rank matrices instead of all the weights
- Linear Algebra — Vectors, matrices, and the language of data
- Neural Networks — Neurons, layers, and how networks learn
- KV Cache — Why LLM generation caches keys and values, and what it costs
- Sampling: Temperature, Top-k & Top-p — How a language model picks its next token
- Mixture of Experts — Routing each token to a few experts in a much larger model
References
- Frantar et al. (2022), "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers"
- Lin et al. (2023), "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration"
- Dettmers et al. (2023), "QLoRA: Efficient Finetuning of Quantized LLMs"
- Dettmers et al. (2022), "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale"
- Nagel et al. (2021), "A White Paper on Neural Network Quantization"
Last updated