Mixture of Experts (MoE) Explained: Routers and Experts

Introduction

DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL has 671 billion parameters, yet uses only 37 billion of them for each token it reads or writes. That is possible because it is a mixture of experts (MoE): instead of one large feed-forward network in each layer, it has many smaller "experts", and a small router sends each token to just a few of them. Many of today's largest open models are built this way, because it buys the knowledge of a huge model at the running cost of a much smaller one. This post explains mixture of experts with diagrams: what replaces what in a Transformer layer, how the router chooses experts, total versus active parameters, why the load on experts must be balanced - with a Java program which routes tokens and balances the load - and the costs that come with MoE.

(Note - The Transformer layer, with its attention and feed-forward blocks, is explained in the Transformer paper postClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained.)

From one feed-forward network to many experts

In a Transformer layer, attention lets tokens exchange information, and a feed-forward network (FFN) then processes each token on its own. The FFN holds two thirds of each layer's parameters - see the GPT-3 parameter countClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners - and much of what the model "knows". A mixture-of-experts layer replaces that one FFN with several, and adds a router -
A dense layer sends every token through one feed-forward network; an MoE layer has a router which scores eight experts and sends the token through the two with the highest scores, whose outputs are mixed by the router's weights Dense layer token vector one big FFNall its weights, every token output Mixture-of-experts layer token vector router: score 8 experts E0 E1 E2 E3 E4 E5 E6 E7 0.53 × E0 + 0.47 × E4mixed by the router's weights only 2 of 8experts run
A dense feed-forward layer and a mixture-of-experts layer, with the routing of token 1 from the program below.
Attention stays shared; only the feed-forward part becomes a set of experts. Each expert is an ordinary feed-forward network of its own. For every token, the router - a single small weight matrix - computes one score per expert, picks the k experts with the highest scores (often 1, 2 or 8), and runs the token through those only. Their outputs are added up, weighted by the router's softmaxClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained over the chosen scores. The other experts do no work for this token.

Total parameters versus active parameters

Because each token passes through only a few experts, an MoE model has two sizes: its total parameters, which hold its knowledge, and its active parameters per token, which decide how much computation each token costs -
Mixtral 8x7B has 46.7 billion parameters and uses 12.9 billion per token; Qwen3-235B-A22B has 235 billion and uses 22 billion; DeepSeek-V3 and R1 have 671 billion and use 37 billion Parameters in billions (bar length to scale) Mixtral 8x7B46.7 total, 12.9 active Qwen3-235B-A22B235 total, 22 active DeepSeek-V3 / R1671 total, 37 active total: stored in memory active: computed per token
Three open MoE models, as published by their developers.
Mixtral 8x7B has 8 experts per layer and uses 2 per token. DeepSeek-V3, the base of R1, has 256 routed experts plus one shared expert in each MoE layer, and uses 8 routed experts per token. In each case, a token costs about as much computation as in a dense model of the active size - the N of the training-compute formula C ≈ 6ND in the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained is the active count - while the model can store the knowledge of one of the total size. The research line goes back to the sparsely-gated MoE layer of Shazeer et al. (2017), and to Google's GShard (2020) and Switch Transformer (2021), which routed each token to a single expert.

Load balancing

A router left to itself tends to favour a few experts: they receive more tokens, train faster, look even better to the router, and receive still more, while other experts sit idle and the hardware holding them waits. MoE training therefore balances the load. Most models add an extra term to the loss which penalizes uneven routing. DeepSeek-V3 instead adds a bias to each expert's score which is used only for choosing experts: after each step, the bias of a busy expert is lowered and that of an idle expert raised, so the routing evens out without an extra loss term. The program below shows both the problem and that remedy.

Routing tokens to experts in Java

The program builds one MoE layer with 8 experts and top-2 routing, using 4-number token vectors so that every step can be printed. It routes three tokens, counts the parameters, and then measures how 4,000 tokens are spread over the experts, without and with bias balancing. Its tokens are deliberately not uniform, as real ones are not, so that the router has favourites.
A mixture-of-experts layer with top-2 routing and bias load balancing
//MixtureOfExperts.java
import java.util.*;

public class MixtureOfExperts {
  static final int DIM = 4, EXPERTS = 8, TOP_K = 2;
  static final Random RANDOM = new Random(11);   // fixed seed, so the output is repeatable

  static double[][] randomMatrix(int rows, int cols) {
    double[][] m = new double[rows][cols];
    for (double[] row : m) for (int j = 0; j < cols; j++) row[j] = RANDOM.nextGaussian() * 0.5;
    return m;
  }

  static double[] multiply(double[][] m, double[] v) {
    double[] out = new double[m.length];
    for (int i = 0; i < m.length; i++) for (int j = 0; j < v.length; j++) out[i] += m[i][j] * v[j];
    return out;
  }

  // Each expert is a small feed-forward network; here one weight matrix keeps the example short
  static final double[][][] EXPERT_WEIGHTS = new double[EXPERTS][][];
  static final double[][] ROUTER = randomMatrix(EXPERTS, DIM);   // one score per expert
  static { for (int e = 0; e < EXPERTS; e++) EXPERT_WEIGHTS[e] = randomMatrix(DIM, DIM); }

  // The router picks the TOP_K experts with the highest scores (scores + per-expert bias for load balancing),
  // and their softmax weights decide how the experts' outputs are mixed
  static int[] route(double[] token, double[] bias, double[] weightsOut) {
    double[] scores = multiply(ROUTER, token);
    Integer[] order = new Integer[EXPERTS];
    for (int e = 0; e < EXPERTS; e++) order[e] = e;
    Arrays.sort(order, (a, b) -> Double.compare(scores[b] + bias[b], scores[a] + bias[a]));
    int[] chosen = new int[TOP_K];
    double sum = 0;
    for (int k = 0; k < TOP_K; k++) { chosen[k] = order[k]; sum += weightsOut[k] = Math.exp(scores[order[k]]); }
    for (int k = 0; k < TOP_K; k++) weightsOut[k] /= sum;
    return chosen;
  }

  static double[] randomToken() {
    double[] t = new double[DIM];
    for (int i = 0; i < DIM; i++) t[i] = RANDOM.nextGaussian() + (i == 0 ? 0.8 : 0);   // real tokens are not uniform
    return t;
  }

  static int[] loads(List<double[]> tokens, double[] bias) {
    int[] load = new int[EXPERTS];
    double[] w = new double[TOP_K];
    for (double[] t : tokens) for (int e : route(t, bias, w)) load[e]++;
    return load;
  }

  public static void main(String[] args) {
    double[] noBias = new double[EXPERTS];
    System.out.println("Routing three tokens (top-" + TOP_K + " of " + EXPERTS + " experts):");
    for (int n = 1; n <= 3; n++) {
      double[] token = randomToken(), w = new double[TOP_K];
      int[] chosen = route(token, noBias, w);
      double[] out = new double[DIM];
      for (int k = 0; k < TOP_K; k++) {
        double[] y = multiply(EXPERT_WEIGHTS[chosen[k]], token);
        for (int i = 0; i < DIM; i++) out[i] += w[k] * y[i];
      }
      System.out.printf("  token %d -> experts %d (weight %.2f) and %d (weight %.2f); output %s%n",
          n, chosen[0], w[0], chosen[1], w[1], Arrays.toString(Arrays.stream(out).map(v -> Math.round(v * 100) / 100.0).toArray()));
    }

    int expertParams = DIM * DIM, routerParams = EXPERTS * DIM;
    System.out.printf("%nParameters: %d in all experts + %d in the router = %d; used per token: %d experts x %d + %d = %d%n",
        EXPERTS * expertParams, routerParams, EXPERTS * expertParams + routerParams,
        TOP_K, expertParams, routerParams, TOP_K * expertParams + routerParams);

    List<double[]> batch = new ArrayList<>();
    for (int i = 0; i < 4000; i++) batch.add(randomToken());
    System.out.println("\nTokens per expert for 4,000 tokens (each token goes to 2 experts; even load = 1,000):");
    System.out.println("  without balancing: " + Arrays.toString(loads(batch, noBias)));

    // Load balancing in the style of DeepSeek-V3: nudge a per-expert bias down for busy experts and up for idle ones.
    // The bias only affects which experts are chosen, not the mixing weights.
    double[] bias = new double[EXPERTS];
    for (int round = 0; round < 200; round++) {
      int[] load = loads(batch, bias);
      for (int e = 0; e < EXPERTS; e++) bias[e] += load[e] > batch.size() * TOP_K / EXPERTS ? -0.01 : 0.01;
    }
    System.out.println("  with bias balancing: " + Arrays.toString(loads(batch, bias)));
  }
}
 OUTPUT of the above code
Routing three tokens (top-2 of 8 experts):
  token 1 -> experts 0 (weight 0.53) and 4 (weight 0.47); output [0.08, -0.04, -0.2, 0.14]
  token 2 -> experts 4 (weight 0.60) and 0 (weight 0.40); output [-0.41, -0.92, -0.8, 1.7]
  token 3 -> experts 0 (weight 0.50) and 7 (weight 0.50); output [-0.3, 1.77, -0.52, 0.04]

Parameters: 128 in all experts + 32 in the router = 160; used per token: 2 experts x 16 + 32 = 64

Tokens per expert for 4,000 tokens (each token goes to 2 experts; even load = 1,000):
  without balancing: [2114, 611, 376, 1320, 1065, 366, 1536, 612]
  with bias balancing: [1011, 1005, 991, 1005, 1007, 992, 993, 996]

Explanation of the code

  • Each token went to two experts, and the router's softmax weights decided how to mix them: token 1 was 0.53 of expert 0 and 0.47 of expert 4. Different tokens chose different pairs.
  • The layer has 160 parameters, but each token used only 64 of them: two experts of 16 plus the router. Scaled up, this is how 671 billion parameters cost 37 billion per token.
  • Without balancing, expert 0 received 2,114 of the 8,000 routing slots and experts 2 and 5 fewer than 400 each - in a real model, the hardware holding experts 2 and 5 would be mostly idle, and expert 0 a bottleneck.
  • After 200 rounds of nudging each expert's bias by 0.01, every expert received between 991 and 1,011 tokens. The bias changed only which experts were chosen; the mixing weights still came from the router's scores.
Without balancing, expert 0 gets 2,114 tokens and experts 2 and 5 under 400; with bias balancing every expert gets between 991 and 1,011 Without balancing even 21140 6111 3762 13203 10654 3665 15366 6127 With bias balancing even 10110 10051 9912 10053 10074 9925 9936 9967 Tokens per expert (0-7) for 4,000 tokens, two experts each; the dashed line is an even share of 1,000.
The load on the eight experts from the program's output, before and after bias balancing.

The costs of a mixture of experts

  • Memory follows the total size. Every expert must be loaded, because any token may need any expert. A 671-billion-parameter MoE needs the memory of a 671-billion-parameter model, even though it computes like a much smaller one. This matters most when you run open-weight models yourself.
  • Communication. In large deployments, experts sit on different GPUs, and tokens must be sent to their experts and back in every MoE layer.
  • Training and fine-tuning are harder. Routing must stay balanced, and training can be less stable than for dense models.
  • Experts are not subject specialists. Despite the name, research on trained MoE models has found that experts do not neatly divide into topics such as "maths" or "biology"; routing often follows patterns in tokens and syntax rather than in subjects.
For an application developer, the practical consequences are on the bill and the hardware: an MoE model behind an API costs and responds roughly like a model of its active size, while running it yourself needs memory for its total size.

Summary

In this post we went through the mixture-of-experts architecture. We saw that an MoE layer replaces a Transformer layer's feed-forward network with many expert networks and a router which sends each token to its top few experts, mixing their outputs by the router's weights. This splits a model's size into total parameters, which set its memory and knowledge, and active parameters, which set the cost per token - 671 and 37 billion for DeepSeek-V3. We saw why routing must be balanced, and in Java we routed tokens through a top-2 layer and evened out a lopsided load with DeepSeek-V3's bias method. Lastly, we went through the costs: memory, communication and harder training.