LLM Quantization Explained: 8-bit, 4-bit and GGUF
Introduction
A 70-billion-parameter model needs 140 GB of memory just for its weights at 16 bits per weight - more than any single consumer GPU, and more than most servers. Stored at 4 bits, the same model needs about 35 GB. That is quantization: storing a model's weights with fewer bits, so that it needs less memory, runs faster and costs less, at the price of a little accuracy. It is why open-weight models run on laptops, and why model downloads come in versions with names such as Q4_K_M or 8-bit (decoded below). This post explains LLM quantization with diagrams: number formats and their memory, how weights are mapped to small integers - implemented in Java - why outliers make this hard and how per-group scales fix it, the common methods and file formats, and what to choose.(Note - Quantization also made fine-tuning cheaper: QLoRA combines a 4-bit model with the adapters explained in the LoRA paper postClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.)
Bits per weight and memory
A model is mostly a huge list of weights - numbers learned in training. How many bytes each one takes decides how much memory the model needs -Memory for a 70B model's weights, as computed by the program below.
Mapping weights to small integers
The simplest method, absmax quantization, maps the weights onto a grid of integers. For 8 bits, the integers run from -127 to 127. A scale is chosen so that the largest weight, in absolute value, lands on 127 -scale = max|w| / 127, q = round(w / scale), w' = q × scaleq × scale. The restored weight w' is close to w, but not equal - the difference is the quantization error. With 4 bits, the grid has only 15 values, from -7 to 7, so each step is much coarser -
Why one large weight ruins a single scale, and how per-group scales contain the damage.
One block of 64 weights, 4-bit, as in the program below. GGUF's quantization types store blocks in a similar layout.
Quantizing weights in Java
The program below quantizes 4,096 weights of a typical size, plus one outlier of 0.9, and compares the formats: how many bytes each weight takes, the average error, and the result of a dot product with an input vector - the basic operation of a model's layers.Absmax quantization to 8 and 4 bits, with one scale and with per-group scales
//Quantization.java
import java.util.*;
public class Quantization {
// Symmetric "absmax" quantization: scale = max|w| / qmax, q = round(w / scale), w' = q * scale
static float[] quantize(float[] w, int bits, int groupSize, int[] qOut) {
int qmax = (1 << (bits - 1)) - 1; // 127 for 8 bits, 7 for 4 bits
float[] restored = new float[w.length];
for (int start = 0; start < w.length; start += groupSize) {
int end = Math.min(start + groupSize, w.length);
float absMax = 0;
for (int i = start; i < end; i++) absMax = Math.max(absMax, Math.abs(w[i]));
float scale = absMax / qmax; // one scale per group, stored next to the integers
for (int i = start; i < end; i++) {
int q = Math.round(w[i] / scale);
if (qOut != null) qOut[i] = q;
restored[i] = q * scale;
}
}
return restored;
}
static double meanAbsError(float[] a, float[] b) {
double e = 0;
for (int i = 0; i < a.length; i++) e += Math.abs(a[i] - b[i]);
return e / a.length;
}
static double dot(float[] a, float[] b) {
double s = 0;
for (int i = 0; i < a.length; i++) s += a[i] * b[i];
return s;
}
public static void main(String[] args) {
Random random = new Random(5);
int n = 4096;
float[] weights = new float[n], input = new float[n];
for (int i = 0; i < n; i++) {
weights[i] = (float) (random.nextGaussian() * 0.02); // typical small weights
input[i] = (float) random.nextGaussian();
}
weights[100] = 0.9f; // one outlier, as real models have
System.out.println("First 6 weights and their 8-bit codes (one scale for all weights):");
int[] codes = new int[n];
float[] w8 = quantize(weights, 8, n, codes);
for (int i = 0; i < 6; i++) System.out.printf(" %+.5f -> %4d -> %+.5f%n", weights[i], codes[i], w8[i]);
System.out.println();
System.out.printf("%-34s %-14s %-12s %s%n", "Format", "bytes/weight", "mean error", "dot product with input");
System.out.printf("%-34s %-14s %-12s %.4f%n", "32-bit floats (original)", "4", "-", dot(weights, input));
Object[][] formats = {{"8-bit, one scale", 8, n}, {"4-bit, one scale", 4, n}, {"4-bit, one scale per 64 weights", 4, 64}};
for (Object[] f : formats) {
int bits = (int) f[1], group = (int) f[2];
float[] q = quantize(weights, bits, group, null);
double bytes = bits / 8.0 + (n / group) * 2.0 / n; // the integers plus a 16-bit scale per group
System.out.printf("%-34s %-14.3f %-12.6f %.4f%n", (String) f[0], bytes, meanAbsError(weights, q), dot(q, input));
}
System.out.println();
System.out.println("Memory for the weights of a 70-billion-parameter model:");
for (Object[] f : new Object[][] {{"32-bit", 4.0}, {"16-bit (BF16)", 2.0}, {"8-bit", 1.0}, {"4-bit", 0.5}})
System.out.printf(" %-14s %5.0f GB%n", f[0], 70e9 * (double) f[1] / 1e9);
}
}
First 6 weights and their 8-bit codes (one scale for all weights): +0.00470 -> 1 -> +0.00709 -0.01777 -> -3 -> -0.02126 -0.00952 -> -1 -> -0.00709 -0.01602 -> -2 -> -0.01417 +0.00724 -> 1 -> +0.00709 +0.01983 -> 3 -> +0.02126 Format bytes/weight mean error dot product with input 32-bit floats (original) 4 - 1.9369 8-bit, one scale 1.000 0.001764 2.0556 4-bit, one scale 0.500 0.016141 0.2542 4-bit, one scale per 64 weights 0.531 0.002126 1.8533 Memory for the weights of a 70-billion-parameter model: 32-bit 280 GB 16-bit (BF16) 140 GB 8-bit 70 GB 4-bit 35 GB
Explanation of the code
- Because of the outlier, the 8-bit scale is 0.9 / 127 = 0.00709, so the first weights became the codes 1, -3, -1, -2: the ordinary weights use only a few of the 255 available values, and are restored with visible error (0.00470 came back as 0.00709).
- With 8 bits and one scale, the dot product moved from 1.9369 to 2.0556. With 4 bits and one scale, the grid step became 0.13, nearly every ordinary weight rounded to 0, and the dot product collapsed to 0.2542 - the layer would no longer compute anything useful.
- With 4 bits and one scale per 64 weights, the average error fell from 0.016141 to 0.002126 and the dot product came back to 1.8533, close to the original, for 0.531 bytes per weight instead of 0.5 - the cost of storing one 16-bit scale per group.
- The memory table is simple multiplication, but it is the reason quantization matters: 140 GB at 16 bits becomes 35 GB at 4 bits.
Methods and formats you will meet
| Name | What it is |
|---|---|
| LLM.int8() (2022) | 8-bit inference which keeps the few outlier feature dimensions in 16 bits and quantizes the rest |
| GPTQ (2022) | Post-training 3- and 4-bit quantization which corrects the error of each weight using the weights not yet quantized, measured on a small calibration dataset |
| AWQ (2023) | Activation-aware 4-bit quantization: protects the small share of weights that matter most for typical activations by scaling them before quantizing |
| NF4 and QLoRA (2023) | A 4-bit format designed for normally distributed weights, used to fine-tune a frozen 4-bit model with LoRA adapters |
| GGUF | The model file format of llama.cpp and the tools built on it, with quantization types such as Q8_0, Q5_K_M and Q4_K_M (roughly: bits per weight, the block-scale scheme, and a size variant) |
| FP8, FP4 | Low-precision floating-point formats with hardware support in recent GPUs, used for inference and increasingly for training |
Most of these are post-training quantization: the model is trained in 16 bits and quantized afterwards, often with a small calibration dataset. Quantization-aware training instead simulates the low precision during training, so the model learns to cope with it; it gives better results at very low bit widths, but needs a training run.
What to choose
- 8 bits is close to lossless for most models, and halves the memory of 16-bit weights.
- 4 bits with per-group scales is the common choice for running open-weight models locally: it usually costs little quality for large models, while smaller models tend to lose more.
- Below 4 bits, quality usually drops quickly; check it on your own tasks before relying on it.
- Measure on your task. Average benchmark scores can hide losses in exact tasks such as arithmetic, code or long-context retrieval, which are often the first to suffer.
- The KV cache counts too. While generating, the model keeps the keys and values of every earlier token, so that attentionClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained does not recompute them. For long contexts (see the context windowClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations) this cache can need as much memory as the weights; many inference servers can quantize it as well.
- Mixture-of-experts models need memory for all their experts, not only the active ones (see mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts), so quantization is what lets a model with hundreds of billions of parameters fit on fewer GPUs.
Summary
In this post we went through LLM quantization. We saw that the bytes per weight decide a model's memory - 140 GB for a 70B model at 16 bits, 35 GB at 4 bits - and that fewer bytes also make generation faster. We went through absmax quantization, which maps weights onto an integer grid with a scale, and in Java we saw an outlier ruin 4-bit quantization with a single scale, and per-group scales repair it at almost no extra cost. Lastly, we went through the methods and formats you will meet - LLM.int8(), GPTQ, AWQ, NF4, GGUF and FP8 - and how to choose a bit width. In the next post, we will look at the other way of making a model cheaper to run, knowledge distillationClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models.Continue Reading: AI and LLM Papers Explained
Knowledge Distillation Explained: Teacher and Student ModelsClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models Mixture of Experts (MoE) Explained: Routers and ExpertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
Knowledge Distillation Explained: Teacher and Student ModelsClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models Mixture of Experts (MoE) Explained: Routers and ExpertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained