LLM Quantization Explained: 8-bit, 4-bit and GGUF

Introduction

A 70-billion-parameter model needs 140 GB of memory just for its weights at 16 bits per weight - more than any single consumer GPU, and more than most servers. Stored at 4 bits, the same model needs about 35 GB. That is quantization: storing a model's weights with fewer bits, so that it needs less memory, runs faster and costs less, at the price of a little accuracy. It is why open-weight models run on laptops, and why model downloads come in versions with names such as Q4_K_M or 8-bit (decoded below). This post explains LLM quantization with diagrams: number formats and their memory, how weights are mapped to small integers - implemented in Java - why outliers make this hard and how per-group scales fix it, the common methods and file formats, and what to choose.

(Note - Quantization also made fine-tuning cheaper: QLoRA combines a 4-bit model with the adapters explained in the LoRA paper postClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.)

Bits per weight and memory

A model is mostly a huge list of weights - numbers learned in training. How many bytes each one takes decides how much memory the model needs -
32-bit floats take 4 bytes per weight, 280 GB for a 70B model; 16-bit 2 bytes, 140 GB; 8-bit 1 byte, 70 GB; 4-bit half a byte, 35 GB Weights of a 70-billion-parameter model 32-bit float4 bytes280 GB 16-bit (BF16, FP16)2 bytes140 GB - how most models are released 8-bit integer1 byte70 GB 4-bit integer½ byte35 GB - fits on one 48 GB GPU, or a large-memory laptop Weights only; running the model also needs memory for the KV cache and activations.
Memory for a 70B model's weights, as computed by the program below.
Fewer bytes help twice. The model fits on smaller and fewer GPUs, and it runs faster, because generating each token means reading every weight from memory once. When a model serves a few requests at a time, as on a laptop or a small server, the speed of memory rather than of arithmetic is the bottleneck, so moving half as many bytes produces tokens noticeably faster.

Mapping weights to small integers

The simplest method, absmax quantization, maps the weights onto a grid of integers. For 8 bits, the integers run from -127 to 127. A scale is chosen so that the largest weight, in absolute value, lands on 127 -
scale = max|w| / 127, q = round(w / scale), w' = q × scale
Only the integers q and the scale are stored (4-bit integers are packed two to a byte); when the model runs, each weight is restored as q × scale. The restored weight w' is close to w, but not equal - the difference is the quantization error. With 4 bits, the grid has only 15 values, from -7 to 7, so each step is much coarser -
With an outlier of 0.9, the 4-bit grid steps are 0.13 apart, so almost all ordinary weights, which lie within about plus or minus 0.05, round to 0; with a separate scale per group of 64 weights, the grid fits each group's own range One scale for all weights (4-bit) -707 almost all weights (blue) round to 0 outlier 0.9 sets the scale One scale per group of 64 weights (4-bit) an ordinary group: the grid spans about ±0.05 the outlier's group: the grid spans ±0.9 The outlier now coarsens only its own 64 weights; every other group keeps a fine grid.
Why one large weight ruins a single scale, and how per-group scales contain the damage.
That is the main difficulty of quantization: outliers. Trained models contain a few weights, and especially activations, that are far larger than the rest. With one scale for a whole matrix, an outlier stretches the grid, and ordinary weights collapse onto a few values, or onto zero. Modern methods therefore use per-group scales - one scale for every block of, say, 32, 64 or 128 weights - so an outlier only coarsens its own block. The extra scales cost a few percent more memory. This is how one block of 64 weights is stored -
64 weights take 128 bytes as 16-bit numbers, and 34 bytes as a 4-bit block: a 2-byte scale and 32 bytes of codes, two codes to a byte 16-bit weights 64 weights x 2 bytes = 128 bytes 4-bit block 2 bytes scale + 32 bytes of codes = 34 bytes: 4.25 bits per weight Inside the codes two codes per byte q0q1q2q3 q4q5q62q63 ... byte 0byte 1 byte 2byte 31 Each code q is an integer from -7 to 7. The weight is restored as q x scale, using the scale of its own block.
One block of 64 weights, 4-bit, as in the program below. GGUF's quantization types store blocks in a similar layout.
Most of these schemes quantize only the weights. At run time the integers are turned back into 16-bit numbers - often on the fly, inside the GPU kernel which does the multiplication - and the arithmetic itself still runs in 16 bits. Quantizing the activations as well (the numbers flowing between layers) would let the GPU use its faster integer units, but activations have even stronger outliers than weights, which makes it harder.

Quantizing weights in Java

The program below quantizes 4,096 weights of a typical size, plus one outlier of 0.9, and compares the formats: how many bytes each weight takes, the average error, and the result of a dot product with an input vector - the basic operation of a model's layers.
Absmax quantization to 8 and 4 bits, with one scale and with per-group scales
//Quantization.java
import java.util.*;

public class Quantization {

  // Symmetric "absmax" quantization: scale = max|w| / qmax, q = round(w / scale), w' = q * scale
  static float[] quantize(float[] w, int bits, int groupSize, int[] qOut) {
    int qmax = (1 << (bits - 1)) - 1;                       // 127 for 8 bits, 7 for 4 bits
    float[] restored = new float[w.length];
    for (int start = 0; start < w.length; start += groupSize) {
      int end = Math.min(start + groupSize, w.length);
      float absMax = 0;
      for (int i = start; i < end; i++) absMax = Math.max(absMax, Math.abs(w[i]));
      float scale = absMax / qmax;                           // one scale per group, stored next to the integers
      for (int i = start; i < end; i++) {
        int q = Math.round(w[i] / scale);
        if (qOut != null) qOut[i] = q;
        restored[i] = q * scale;
      }
    }
    return restored;
  }

  static double meanAbsError(float[] a, float[] b) {
    double e = 0;
    for (int i = 0; i < a.length; i++) e += Math.abs(a[i] - b[i]);
    return e / a.length;
  }

  static double dot(float[] a, float[] b) {
    double s = 0;
    for (int i = 0; i < a.length; i++) s += a[i] * b[i];
    return s;
  }

  public static void main(String[] args) {
    Random random = new Random(5);
    int n = 4096;
    float[] weights = new float[n], input = new float[n];
    for (int i = 0; i < n; i++) {
      weights[i] = (float) (random.nextGaussian() * 0.02);   // typical small weights
      input[i] = (float) random.nextGaussian();
    }
    weights[100] = 0.9f;                                       // one outlier, as real models have

    System.out.println("First 6 weights and their 8-bit codes (one scale for all weights):");
    int[] codes = new int[n];
    float[] w8 = quantize(weights, 8, n, codes);
    for (int i = 0; i < 6; i++) System.out.printf("  %+.5f -> %4d -> %+.5f%n", weights[i], codes[i], w8[i]);

    System.out.println();
    System.out.printf("%-34s %-14s %-12s %s%n", "Format", "bytes/weight", "mean error", "dot product with input");
    System.out.printf("%-34s %-14s %-12s %.4f%n", "32-bit floats (original)", "4", "-", dot(weights, input));
    Object[][] formats = {{"8-bit, one scale", 8, n}, {"4-bit, one scale", 4, n}, {"4-bit, one scale per 64 weights", 4, 64}};
    for (Object[] f : formats) {
      int bits = (int) f[1], group = (int) f[2];
      float[] q = quantize(weights, bits, group, null);
      double bytes = bits / 8.0 + (n / group) * 2.0 / n;      // the integers plus a 16-bit scale per group
      System.out.printf("%-34s %-14.3f %-12.6f %.4f%n", (String) f[0], bytes, meanAbsError(weights, q), dot(q, input));
    }

    System.out.println();
    System.out.println("Memory for the weights of a 70-billion-parameter model:");
    for (Object[] f : new Object[][] {{"32-bit", 4.0}, {"16-bit (BF16)", 2.0}, {"8-bit", 1.0}, {"4-bit", 0.5}})
      System.out.printf("  %-14s %5.0f GB%n", f[0], 70e9 * (double) f[1] / 1e9);
  }
}
 OUTPUT of the above code
First 6 weights and their 8-bit codes (one scale for all weights):
  +0.00470 ->    1 -> +0.00709
  -0.01777 ->   -3 -> -0.02126
  -0.00952 ->   -1 -> -0.00709
  -0.01602 ->   -2 -> -0.01417
  +0.00724 ->    1 -> +0.00709
  +0.01983 ->    3 -> +0.02126

Format                             bytes/weight   mean error   dot product with input
32-bit floats (original)           4              -            1.9369
8-bit, one scale                   1.000          0.001764     2.0556
4-bit, one scale                   0.500          0.016141     0.2542
4-bit, one scale per 64 weights    0.531          0.002126     1.8533

Memory for the weights of a 70-billion-parameter model:
  32-bit           280 GB
  16-bit (BF16)    140 GB
  8-bit             70 GB
  4-bit             35 GB

Explanation of the code

  • Because of the outlier, the 8-bit scale is 0.9 / 127 = 0.00709, so the first weights became the codes 1, -3, -1, -2: the ordinary weights use only a few of the 255 available values, and are restored with visible error (0.00470 came back as 0.00709).
  • With 8 bits and one scale, the dot product moved from 1.9369 to 2.0556. With 4 bits and one scale, the grid step became 0.13, nearly every ordinary weight rounded to 0, and the dot product collapsed to 0.2542 - the layer would no longer compute anything useful.
  • With 4 bits and one scale per 64 weights, the average error fell from 0.016141 to 0.002126 and the dot product came back to 1.8533, close to the original, for 0.531 bytes per weight instead of 0.5 - the cost of storing one 16-bit scale per group.
  • The memory table is simple multiplication, but it is the reason quantization matters: 140 GB at 16 bits becomes 35 GB at 4 bits.

Methods and formats you will meet

NameWhat it is
LLM.int8() (2022)8-bit inference which keeps the few outlier feature dimensions in 16 bits and quantizes the rest
GPTQ (2022)Post-training 3- and 4-bit quantization which corrects the error of each weight using the weights not yet quantized, measured on a small calibration dataset
AWQ (2023)Activation-aware 4-bit quantization: protects the small share of weights that matter most for typical activations by scaling them before quantizing
NF4 and QLoRA (2023)A 4-bit format designed for normally distributed weights, used to fine-tune a frozen 4-bit model with LoRA adapters
GGUFThe model file format of llama.cpp and the tools built on it, with quantization types such as Q8_0, Q5_K_M and Q4_K_M (roughly: bits per weight, the block-scale scheme, and a size variant)
FP8, FP4Low-precision floating-point formats with hardware support in recent GPUs, used for inference and increasingly for training
The same idea shrinks embedding indexes: vector stores can keep each stored vector with 8 bits per number, or even 1 bit, as described in the vector store comparisonClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared.

Most of these are post-training quantization: the model is trained in 16 bits and quantized afterwards, often with a small calibration dataset. Quantization-aware training instead simulates the low precision during training, so the model learns to cope with it; it gives better results at very low bit widths, but needs a training run.

What to choose

With hosted APIs, quantization is the provider's decision and is usually not visible; it matters when you run models yourself, where it decides which model fits on which hardware. Whether to run a model yourself at all is the subject of open-weight vs closed-weight vs open-source LLMsClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained.

Summary

In this post we went through LLM quantization. We saw that the bytes per weight decide a model's memory - 140 GB for a 70B model at 16 bits, 35 GB at 4 bits - and that fewer bytes also make generation faster. We went through absmax quantization, which maps weights onto an integer grid with a scale, and in Java we saw an outlier ruin 4-bit quantization with a single scale, and per-group scales repair it at almost no extra cost. Lastly, we went through the methods and formats you will meet - LLM.int8(), GPTQ, AWQ, NF4, GGUF and FP8 - and how to choose a bit width. In the next post, we will look at the other way of making a model cheaper to run, knowledge distillationClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models.