Mixture of Experts (MoE) Explained: How Sparse AI Models Work

In short: a mixture-of-experts (MoE) model splits part of its network into many smaller sub-networks called experts, and a small router picks only a few of them for each token. The model can hold a very large number of parameters while using only a fraction of them for any one word it reads or writes. The idea was scaled up for neural networks by Google researchers in 2017, and many large open-weight models today, from DeepSeek, Xiaomi, Meta and Mistral among others, publish two parameter counts for this reason: total and active.

What is a mixture of experts, in simple words?

A mixture of experts is a model that only switches on the parts of itself it needs for each piece of text. A normal ("dense") model runs every one of its parameters for every token. An MoE model replaces some of its layers with a set of experts and a router. For each token, the router scores the experts and sends the token to the best few; the others sit idle for that token.

A rough picture: a dense model is one generalist who reads every question in full. An MoE model is a large office of specialists with a receptionist who passes each question to two or three of them. The office is much bigger, but each question only takes a few people's time.

For how tokens and layers work in the first place, see how LLMs work.

Total parameters vs active parameters: what is the difference?

Total parameters is everything the model contains; active parameters is how much of it runs for each token. Both numbers matter, for different reasons:

Number	What it measures	What it affects
Total parameters	All weights, including every expert	How much memory the model needs, because every expert has to be loaded
Active parameters	The weights used for one token	How much computing each token costs, and so speed and running cost

Published examples from vendors' own model cards:

Model	Total	Active per token	Source
Mixtral 8x7B (Mistral, 2024)	47B	13B	Mixtral paper
DeepSeek-V3 (2024)	671B	37B	DeepSeek-V3 technical report
MiMo-V2.6-Pro (Xiaomi)	1.02T	42B	Hugging Face model card
Muse Spark Open (Meta)	816B	42B	Meta developer docs
MiMo-V2.6-Flash (Xiaomi)	309B	15B	Hugging Face model card
DeepSeek V4.1 Flash	552B	8B when reading the prompt, 16B when generating	Hugging Face model card

So "a 1-trillion-parameter model" and "a 42-billion-parameter model" can describe the same MoE model, depending on which count is quoted.

How does the router choose experts?

The router is a small trainable network that scores every expert for the current token and keeps the top few. In Mistral's description of Mixtral 8x7B, each layer has 8 feed-forward blocks (the experts); "for every token, at each layer, a router network selects two experts to process the current state and combine their outputs." The chosen pair can be different at every layer and for every token.

Three details beginners usually miss:

• Routing happens per token, not per question. One sentence can pass through many different experts.
• Routing happens per layer. A token that uses experts 2 and 5 in one layer may use 1 and 7 in the next.
• Only some layers are experts. In Mixtral, the experts replace the feed-forward part of each layer; the attention part is shared by every token. See what attention is.

Do the experts specialise in topics?

Not in the way the name suggests. The experts are not hand-assigned subjects like "maths" or "French"; they are learned during training, and what each one ends up handling is whatever reduced the training loss. The Mixtral paper's routing analysis reported that it did not observe obvious patterns in how experts were assigned based on the topic of the text, such as biology papers versus philosophy papers. What it did see was some structure tied to syntax, and consecutive tokens often going to the same experts.

Where did mixture of experts come from?

The approach was scaled up for modern neural networks in a 2017 Google paper, and its routing was simplified in 2021.

• 2017 - Sparsely-gated MoE. Outrageously Large Neural Networks by Noam Shazeer and colleagues at Google introduced a layer of "up to thousands of feed-forward sub-networks" with "a trainable gating network". The authors reported "greater than 1000x improvements in model capacity with only minor losses in computational efficiency", in models of up to 137 billion parameters.
• 2021 - Switch Transformer. Switch Transformers (Fedus, Zoph and Shazeer, Google) simplified routing so that each token goes to a single expert, and pre-trained models of up to a trillion parameters. The paper reported up to a 7x increase in pre-training speed over T5-Base with the same computing resources.
• 2024 - Open-weight MoE. Mistral released Mixtral 8x7B under the Apache 2.0 licence, and DeepSeek's V3 report described a 671B-total, 37B-active model. Since then, total-and-active parameter counts have become a standard line on open-weight model cards.

The general idea is older: the term "mixture of experts" dates to research in the early 1990s. The 2017 paper's contribution was making it work at the scale of large language models.

Why do model makers use mixture of experts?

Because it lets a model grow its total size without the cost per token growing at the same rate. The Switch Transformer paper describes the result as "a sparsely-activated model - with outrageous numbers of parameters - but a constant computational cost". The trade-offs the same papers name:

• Memory. Every expert must be in memory even though only a few run per token, so an MoE model needs hardware sized for its total parameters. This is why running a large MoE model locally is harder than its active count suggests; see how to run models locally.
• Load balancing. If the router sends most tokens to a few favourite experts, the rest are wasted. Training usually adds an extra loss term to spread tokens evenly; the DeepSeek-V3 report describes "an auxiliary-loss-free strategy for load balancing" instead.
• Training stability and communication. The Switch Transformer paper names "complexity, communication costs and training instability" as what had held MoE back, since experts are often spread across many chips that must exchange tokens.

How do MoE models rank against dense models?

Architecture alone does not decide a model's rank; MoE and dense models sit side by side on the same leaderboards. On LMArena's text leaderboard, DeepSeek V4.1 Flash holds 1475 Elo (rank 38, max effort) and MiMo-V2.6-Pro holds 1479 Elo (rank 27). The full table is on the AI model rankings page. For what Elo and benchmarks each measure, see how to compare AI models.

Words to know

• Expert: one of many small feed-forward sub-networks inside an MoE layer.
• Router (gating network): the small network that scores experts and picks which ones each token uses.
• Top-k routing: sending each token to the k highest-scoring experts; Mixtral uses k = 2, Switch Transformer uses k = 1.
• Dense model: a model that uses all of its parameters for every token.
• Sparse model: a model that uses only part of its parameters for each token; MoE is the common kind.
• Total parameters: every weight in the model, all experts included.
• Active parameters: the weights used to process one token.
• Load balancing: keeping tokens spread across experts so none sit unused.

Related guides

• How LLMs work
• Scaling laws: Kaplan and Chinchilla explained
• Attention Is All You Need: the transformer paper
• How to run AI models locally