LoRA Paper Explained: Low-Rank Adaptation of LLMs
Introduction
To specialize a pre-trained language model for a task - answering support tickets in a company's style, say - the classic approach, popularized by BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers, is full fine-tuning: continue training all of its weights on task data. For a model the size of GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners, that means updating 175 billion parameters, holding their gradients and optimizer state in GPU memory, and saving a new 350 GB copy of the model for every task. The 2021 paper LoRA: Low-Rank Adaptation of Large Language Models, from Microsoft, showed that you can instead freeze the model and train two small matrices per layer, cutting the trainable parameters by up to 10,000 times while matching the quality of full fine-tuning. This post explains the LoRA paper with diagrams. We will see the low-rank idea behind it, how the LoRA matrices are initialized, scaled and merged, which weights to adapt, and the results - with Java code that counts the trainable parameters and merges an adapter.The paper: LoRA: Low-Rank Adaptation of Large Language Models
Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (Microsoft)
Published: June 2021 - ICLR 2022
Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (Microsoft)
Published: June 2021 - ICLR 2022
The idea: weight updates have low rank
Fine-tuning changes a weight matrixW0 into W0 + ΔW. In full fine-tuning, the update ΔW is as big as W0 itself - for a 12,288 × 12,288 matrix in GPT-3, about 151 million numbers. Earlier research had found that pre-trained models have a low intrinsic dimension: they can be adapted well while changing them in only a small number of directions. The LoRA authors hypothesized that the update ΔW therefore has a low rank. The rank of a matrix is the number of independent directions its rows and columns span; a matrix of rank r, however large, can always be written as the product of two thin matrices with r columns and r rows -
ΔW = B · A, where B is d × r, A is r × k, and the rank r is much smaller than d and kFor one 12,288 × 12,288 matrix of GPT-3, a rank-4 update needs 0.07% of the numbers of a full update.
How a LoRA layer works
During fine-tuning, the original weightsW0 are frozen - they receive no gradient updates. Only A and B are trained. The layer's output becomes the sum of the frozen path and the low-rank path -
h = W0x + (α / r) · B A xA LoRA layer: the frozen path and the trainable low-rank path run side by side.
- Initialization:
Astarts with small random values andBstarts at zero, soBA = 0at the start. The model therefore begins fine-tuning behaving exactly like the pre-trained model. - Scaling: the low-rank output is multiplied by
α / r, whereαis a constant. This keeps the size of the update roughly the same when you try different ranks, so other settings such as the learning rate do not need to be re-tuned. - Merging: after training,
BAcan be added intoW0to give a single matrixW = W0 + BA. The merged model has exactly the original size and speed - LoRA adds no inference latency, unlike earlier "adapter" methods which inserted extra layers. To switch tasks, subtract oneBAand add another.
Counting parameters and merging in Java
The program below counts the trainable parameters when LoRA is applied to the query and value matrices of all 96 layers of GPT-3, and then checks, on a small example, that mergingBA into the weights gives exactly the same output as keeping the two paths separate.
Trainable parameters and merging a LoRA adapter
//LoraDemo.java
import java.util.Arrays;
import java.util.stream.Collectors;
public class LoraDemo {
static double[][] matmul(double[][] a, double[][] b) {
double[][] out = new double[a.length][b[0].length];
for (int i = 0; i < a.length; i++)
for (int j = 0; j < b[0].length; j++)
for (int k = 0; k < b.length; k++) out[i][j] += a[i][k] * b[k][j];
return out;
}
static String format(double[][] column) {
return Arrays.stream(column).map(row -> String.format("%.4f", row[0])).collect(Collectors.joining(", ", "[", "]"));
}
static double[][] add(double[][] a, double[][] b) {
double[][] out = new double[a.length][a[0].length];
for (int i = 0; i < a.length; i++)
for (int j = 0; j < a[0].length; j++) out[i][j] = a[i][j] + b[i][j];
return out;
}
public static void main(String[] args) {
// 1. Trainable parameters for GPT-3 175B: d = 12,288, 96 layers, LoRA on Wq and Wv
long d = 12_288, layers = 96, adaptedPerLayer = 2;
System.out.printf("Full fine-tuning: %,d trainable parameters%n", 174_600_000_000L);
for (int r : new int[] {1, 4, 8}) {
long perMatrix = d * r + r * d; // B is d x r, A is r x d
long total = perMatrix * adaptedPerLayer * layers;
System.out.printf("LoRA r=%d: %,d trainable parameters (%,.0fx fewer)%n", r, total, 174.6e9 / total);
}
// 2. Merging: (W0 + BA) x == W0 x + B (A x)
double[][] w0 = { {0.5, -0.2, 0.1}, {0.3, 0.8, -0.5}, {-0.4, 0.2, 0.9} }; // frozen 3 x 3
double[][] b = { {0.2}, {-0.1}, {0.4} }; // trained 3 x 1
double[][] a = { {1.0, 0.5, -1.0} }; // trained 1 x 3 (rank 1)
double[][] x = { {1.0}, {2.0}, {3.0} };
double[][] separate = add(matmul(w0, x), matmul(b, matmul(a, x)));
double[][] merged = matmul(add(w0, matmul(b, a)), x);
System.out.println("Two paths: " + format(separate));
System.out.println("Merged W : " + format(merged));
System.out.println("Merged weights have the same shape as W0: " + matmul(b, a).length + " x " + matmul(b, a)[0].length);
}
}
Full fine-tuning: 174,600,000,000 trainable parameters LoRA r=1: 4,718,592 trainable parameters (37,003x fewer) LoRA r=4: 18,874,368 trainable parameters (9,251x fewer) LoRA r=8: 37,748,736 trainable parameters (4,625x fewer) Two paths: [0.2000, 0.5000, 2.3000] Merged W : [0.2000, 0.5000, 2.3000] Merged weights have the same shape as W0: 3 x 3
Explanation of the code
- With rank 1 and rank 8, the counts are 4.7 million and 37.7 million - the two LoRA configurations reported for GPT-3 in the paper. Even at rank 8, the trainable parameters are over 4,000 times fewer than for full fine-tuning.
- The merged and separate outputs are the same (apart from floating-point rounding in the last digits, which is why they are printed to 4 decimals), which is why a merged LoRA model runs at exactly the original speed. (The scaling factor
α / ris left out here for simplicity; it would simply multiplyBA.) B · Ais a full 3 × 3 matrix, but it was built from only 6 trained numbers instead of 9. At GPT-3's size, the same trick turns 151 million numbers per matrix into about 25 thousand per unit of rank.
Which weights to adapt, and what rank?
In principle LoRA can be applied to any weight matrix. The paper applied it only to the attention weights of each TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained layer, and studied which ones matter -- With a fixed budget of trainable parameters, adapting both the query (Wq) and value (Wv) matrices worked better than putting the whole budget into one matrix with a higher rank.
- A very small rank was enough: on GPT-3,
r = 1or2on Wq and Wv performed about as well as much larger ranks. The authors also found that the learned updates amplify directions which were already present, but not emphasized, in the pre-trained weights.
Results
- Quality: on RoBERTa, DeBERTa, GPT-2 and GPT-3, LoRA matched or beat full fine-tuning on benchmarks covering natural language understanding, SQL generation from text, and summarization.
- Memory: for GPT-3 175B, GPU memory during training fell from 1.2 TB to 350 GB, because no gradients or optimizer state are needed for the frozen weights.
- Storage: a GPT-3 checkpoint for a task shrank from 350 GB to about 35 MB - roughly 10,000 times smaller - so hundreds of task-specific versions can share one copy of the base model.
- Speed: training was about 25% faster than full fine-tuning, and inference had no extra latency.
LoRA's impact
LoRA made fine-tuning large models practical for anyone with a single GPU, especially once open-weight models such as Llama appeared. QLoRA (2023) combined it with a base model quantized to 4 bits - each weight stored in 4 bits instead of 16 - making it possible to fine-tune a 65-billion-parameter model on one 48 GB GPU. LoRA is now the default form of parameter-efficient fine-tuning (PEFT): model hubs share thousands of small LoRA adapters, and serving systems can keep one base model in memory and switch adapters per request. It is used the same way for image models, to teach them a new style or subject from a handful of examples. Instruction tuning and preference tuning of open models, the steps described in the InstructGPT paperClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, are also commonly done with LoRA adapters.Serving many tasks with LoRA: each request uses its own small adapter on top of one shared copy of the base model.
Summary
In this post we went through the LoRA paper. We saw why full fine-tuning of large models is expensive, and LoRA's hypothesis that weight updates have a low rank, so that each update can be written as the product of two thin matrices, B and A. We went through how a LoRA layer adds a trainable low-rank path beside the frozen weights, with B initialized to zero and the output scaled by α / r, and how the update can be merged into the weights with no extra latency. In Java, we reproduced the paper's trainable-parameter counts for GPT-3 and verified the merge. Lastly, we looked at which weights to adapt, the results, and how LoRA became the standard for parameter-efficient fine-tuning. In the next post, we will see how the right prompt can unlock reasoning, in the chain-of-thought prompting paperClick to read tutorial: Chain-of-Thought Prompting Paper Explained.Continue Reading: AI and LLM Papers Explained
Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL