Mixture of Experts (MoE) Explained: Routers and Experts
Introduction
DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL has 671 billion parameters, yet uses only 37 billion of them for each token it reads or writes. That is possible because it is a mixture of experts (MoE): instead of one large feed-forward network in each layer, it has many smaller "experts", and a small router sends each token to just a few of them. Many of today's largest open models are built this way, because it buys the knowledge of a huge model at the running cost of a much smaller one. This post explains mixture of experts with diagrams: what replaces what in a Transformer layer, how the router chooses experts, total versus active parameters, why the load on experts must be balanced - with a Java program which routes tokens and balances the load - and the costs that come with MoE.(Note - The Transformer layer, with its attention and feed-forward blocks, is explained in the Transformer paper postClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained.)
From one feed-forward network to many experts
In a Transformer layer, attention lets tokens exchange information, and a feed-forward network (FFN) then processes each token on its own. The FFN holds two thirds of each layer's parameters - see the GPT-3 parameter countClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners - and much of what the model "knows". A mixture-of-experts layer replaces that one FFN with several, and adds a router -A dense feed-forward layer and a mixture-of-experts layer, with the routing of token 1 from the program below.
Total parameters versus active parameters
Because each token passes through only a few experts, an MoE model has two sizes: its total parameters, which hold its knowledge, and its active parameters per token, which decide how much computation each token costs -Three open MoE models, as published by their developers.
Load balancing
A router left to itself tends to favour a few experts: they receive more tokens, train faster, look even better to the router, and receive still more, while other experts sit idle and the hardware holding them waits. MoE training therefore balances the load. Most models add an extra term to the loss which penalizes uneven routing. DeepSeek-V3 instead adds a bias to each expert's score which is used only for choosing experts: after each step, the bias of a busy expert is lowered and that of an idle expert raised, so the routing evens out without an extra loss term. The program below shows both the problem and that remedy.Routing tokens to experts in Java
The program builds one MoE layer with 8 experts and top-2 routing, using 4-number token vectors so that every step can be printed. It routes three tokens, counts the parameters, and then measures how 4,000 tokens are spread over the experts, without and with bias balancing. Its tokens are deliberately not uniform, as real ones are not, so that the router has favourites.A mixture-of-experts layer with top-2 routing and bias load balancing
//MixtureOfExperts.java
import java.util.*;
public class MixtureOfExperts {
static final int DIM = 4, EXPERTS = 8, TOP_K = 2;
static final Random RANDOM = new Random(11); // fixed seed, so the output is repeatable
static double[][] randomMatrix(int rows, int cols) {
double[][] m = new double[rows][cols];
for (double[] row : m) for (int j = 0; j < cols; j++) row[j] = RANDOM.nextGaussian() * 0.5;
return m;
}
static double[] multiply(double[][] m, double[] v) {
double[] out = new double[m.length];
for (int i = 0; i < m.length; i++) for (int j = 0; j < v.length; j++) out[i] += m[i][j] * v[j];
return out;
}
// Each expert is a small feed-forward network; here one weight matrix keeps the example short
static final double[][][] EXPERT_WEIGHTS = new double[EXPERTS][][];
static final double[][] ROUTER = randomMatrix(EXPERTS, DIM); // one score per expert
static { for (int e = 0; e < EXPERTS; e++) EXPERT_WEIGHTS[e] = randomMatrix(DIM, DIM); }
// The router picks the TOP_K experts with the highest scores (scores + per-expert bias for load balancing),
// and their softmax weights decide how the experts' outputs are mixed
static int[] route(double[] token, double[] bias, double[] weightsOut) {
double[] scores = multiply(ROUTER, token);
Integer[] order = new Integer[EXPERTS];
for (int e = 0; e < EXPERTS; e++) order[e] = e;
Arrays.sort(order, (a, b) -> Double.compare(scores[b] + bias[b], scores[a] + bias[a]));
int[] chosen = new int[TOP_K];
double sum = 0;
for (int k = 0; k < TOP_K; k++) { chosen[k] = order[k]; sum += weightsOut[k] = Math.exp(scores[order[k]]); }
for (int k = 0; k < TOP_K; k++) weightsOut[k] /= sum;
return chosen;
}
static double[] randomToken() {
double[] t = new double[DIM];
for (int i = 0; i < DIM; i++) t[i] = RANDOM.nextGaussian() + (i == 0 ? 0.8 : 0); // real tokens are not uniform
return t;
}
static int[] loads(List<double[]> tokens, double[] bias) {
int[] load = new int[EXPERTS];
double[] w = new double[TOP_K];
for (double[] t : tokens) for (int e : route(t, bias, w)) load[e]++;
return load;
}
public static void main(String[] args) {
double[] noBias = new double[EXPERTS];
System.out.println("Routing three tokens (top-" + TOP_K + " of " + EXPERTS + " experts):");
for (int n = 1; n <= 3; n++) {
double[] token = randomToken(), w = new double[TOP_K];
int[] chosen = route(token, noBias, w);
double[] out = new double[DIM];
for (int k = 0; k < TOP_K; k++) {
double[] y = multiply(EXPERT_WEIGHTS[chosen[k]], token);
for (int i = 0; i < DIM; i++) out[i] += w[k] * y[i];
}
System.out.printf(" token %d -> experts %d (weight %.2f) and %d (weight %.2f); output %s%n",
n, chosen[0], w[0], chosen[1], w[1], Arrays.toString(Arrays.stream(out).map(v -> Math.round(v * 100) / 100.0).toArray()));
}
int expertParams = DIM * DIM, routerParams = EXPERTS * DIM;
System.out.printf("%nParameters: %d in all experts + %d in the router = %d; used per token: %d experts x %d + %d = %d%n",
EXPERTS * expertParams, routerParams, EXPERTS * expertParams + routerParams,
TOP_K, expertParams, routerParams, TOP_K * expertParams + routerParams);
List<double[]> batch = new ArrayList<>();
for (int i = 0; i < 4000; i++) batch.add(randomToken());
System.out.println("\nTokens per expert for 4,000 tokens (each token goes to 2 experts; even load = 1,000):");
System.out.println(" without balancing: " + Arrays.toString(loads(batch, noBias)));
// Load balancing in the style of DeepSeek-V3: nudge a per-expert bias down for busy experts and up for idle ones.
// The bias only affects which experts are chosen, not the mixing weights.
double[] bias = new double[EXPERTS];
for (int round = 0; round < 200; round++) {
int[] load = loads(batch, bias);
for (int e = 0; e < EXPERTS; e++) bias[e] += load[e] > batch.size() * TOP_K / EXPERTS ? -0.01 : 0.01;
}
System.out.println(" with bias balancing: " + Arrays.toString(loads(batch, bias)));
}
}
Routing three tokens (top-2 of 8 experts): token 1 -> experts 0 (weight 0.53) and 4 (weight 0.47); output [0.08, -0.04, -0.2, 0.14] token 2 -> experts 4 (weight 0.60) and 0 (weight 0.40); output [-0.41, -0.92, -0.8, 1.7] token 3 -> experts 0 (weight 0.50) and 7 (weight 0.50); output [-0.3, 1.77, -0.52, 0.04] Parameters: 128 in all experts + 32 in the router = 160; used per token: 2 experts x 16 + 32 = 64 Tokens per expert for 4,000 tokens (each token goes to 2 experts; even load = 1,000): without balancing: [2114, 611, 376, 1320, 1065, 366, 1536, 612] with bias balancing: [1011, 1005, 991, 1005, 1007, 992, 993, 996]
Explanation of the code
- Each token went to two experts, and the router's softmax weights decided how to mix them: token 1 was 0.53 of expert 0 and 0.47 of expert 4. Different tokens chose different pairs.
- The layer has 160 parameters, but each token used only 64 of them: two experts of 16 plus the router. Scaled up, this is how 671 billion parameters cost 37 billion per token.
- Without balancing, expert 0 received 2,114 of the 8,000 routing slots and experts 2 and 5 fewer than 400 each - in a real model, the hardware holding experts 2 and 5 would be mostly idle, and expert 0 a bottleneck.
- After 200 rounds of nudging each expert's bias by 0.01, every expert received between 991 and 1,011 tokens. The bias changed only which experts were chosen; the mixing weights still came from the router's scores.
The load on the eight experts from the program's output, before and after bias balancing.
The costs of a mixture of experts
- Memory follows the total size. Every expert must be loaded, because any token may need any expert. A 671-billion-parameter MoE needs the memory of a 671-billion-parameter model, even though it computes like a much smaller one. This matters most when you run open-weight models yourself.
- Communication. In large deployments, experts sit on different GPUs, and tokens must be sent to their experts and back in every MoE layer.
- Training and fine-tuning are harder. Routing must stay balanced, and training can be less stable than for dense models.
- Experts are not subject specialists. Despite the name, research on trained MoE models has found that experts do not neatly divide into topics such as "maths" or "biology"; routing often follows patterns in tokens and syntax rather than in subjects.
Summary
In this post we went through the mixture-of-experts architecture. We saw that an MoE layer replaces a Transformer layer's feed-forward network with many expert networks and a router which sends each token to its top few experts, mixing their outputs by the router's weights. This splits a model's size into total parameters, which set its memory and knowledge, and active parameters, which set the cost per token - 671 and 37 billion for DeepSeek-V3. We saw why routing must be balanced, and in Java we routed tokens through a top-2 layer and evened out a lopsided load with DeepSeek-V3's bias method. Lastly, we went through the costs: memory, communication and harder training.Continue Reading: AI and LLM Papers Explained
Softmax, Cross-Entropy and KL Divergence ExplainedClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF
Softmax, Cross-Entropy and KL Divergence ExplainedClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF