Mixture of Experts (MoEs) in Transformers

Over the past few years, scaling dense language models has driven most progress in LLMs. From early models like the original ULMFiT (~30M parameters) or GPT-2 (1.5B parameters, which at the time was considered "too dangerous to release" 🧌), and eventually to today’s hundred-billion–parameter systems, the recipe was simple:

More data + more parameters gives better performance.

Scaling laws reinforced this trend, but dense scaling has practical limits:

• Training becomes increasingly expensive.
• Inference latency grows.
• Deployment requires significant memory and hardware.

This is where Mixture of Experts (MoEs) enter the picture.

If you're already familiar with MoEs and want to jump straight into the engineering work done in transformers, you can head directly to Transformers and MoEs.

A Mixture of Experts model keeps the Transformer backbone, but replaces certain dense feed-forward layers with a set of experts. An “expert” is not a topic-specialized module (e.g., "math expert", "code expert"). It is simply a learnable sub-network. For each token, a router selects a small subset of experts to process it.

Different tokens activate different experts, based on their hidden representations.

Model capacity depends on total parameters, but inference speed depends on active parameters.

This is the key idea.

For example, take gpt-oss-20b

. It has 21B total parameters, but uses 4 active experts per token, out of a total of 32 experts. Considering the shared components plus the active experts, this model uses ~3.6B active parameters per token. Running this model on an M3 Ultra Mac, which has a memory bandwidth of about 800 GB, we could estimate generation speed as ~ 800 / (3.6 * 2)

in bfloat16

, where each parameter takes 2 bytes. This yields about 111 tokens per second. The actual performance number we get is ~115 tok/s, which is very close to the back-of-the-envelope calculation.

This super fast speed confirms the model works approximately as a 3.6B parameter one, but it has the same capacity (or quality) as a 21B parameter model.

(Note: speed would be even faster if we used kernels for the native mxfp4 quantization the model uses).

MoEs are attractive for these reasons:

Better Compute Efficiency

Given a fixed training FLOP budget, MoEs often outperform dense counterparts.

This means faster iteration and better scaling efficiency.

A Natural Parallelization Axis

Experts provide a structural boundary in the computation graph. Since different tokens engage different experts, we can parallelize across experts (we discuss this later in Expert Parallelism).

Industry Adoption

Recent major MoE releases of open models that happened in the past few weeks include Qwen 3.5, MiniMax M2, GLM-5, or Kimi K2.5.

The trend accelerated after the success of DeepSeek R1 in January 2025, building on earlier systems like DeepSeek V2. Another early MoE was Mixtral-8x7B, released in December 2023.

Figure 3: 2-year timeline of MoE model addition to the transformers

library. DeepSeek R1 marks a clear inflection point.Closed labs use MoEs too. ChatGPT has long been

rumoredto use a sparse architecture, and the open gpt-oss models certainly do.

If you want to learn more about MoEs in general, we strongly suggest reading this blog and watching our recent YouTube video on routing.

Most tooling in the ecosystem, including model loading, device placement, quantization, and backend execution was originally designed for dense models. MoEs challenge these assumptions.

Making MoEs first-class citizens in transformers

means redesigning parts of the loading pipeline, execution model, and distributed abstractions, not just adding new model classes. We’ll focus on how the transformers

library has evolved to support sparse architectures across:

AutoModelForCausalLM.from_pretrained("model_id")

downloads and loads model weights into a PyTorch model. For dense models, loading is relatively straightforward where each tensor in the checkpoint maps one-to-one to a parameter in the runtime module.

For MoEs, it’s more complicated. In most MoE checkpoints, each expert is serialized independently. If you peek inside the DeepSeek-V3 checkpoint index, you’ll see keys like:

model.layers.3.mlp.experts.0.gate_proj.weight
...
model.layers.3.mlp.experts.255.gate_proj.weight

Each expert has its own set of weight matrices, essentially 256 (0 to 255 total, taking DeepSeek-V3 as an example) small feed-forward networks saved side by side. At runtime, however, GPUs execute optimized kernels. Modern MoE kernels such as grouped GEMMs and fused MoE implementations are designed to process all experts in a single operation, not by looping over them one at a time.

To do that efficiently, they require expert weights to be packed into a single contiguous tensor.

So we have a mismatch:

**Checkpoint:256 separate tensorsRuntime:**1 packed tensor

Bridging this gap systematically is what the weight loading refactor enables.

With the introduction of a generic WeightConverter, the mental model shifted from:

A checkpoint already matches my runtime layout; loading is mostly a key-by-key copy.

to:

A checkpoint is just a serialized source of tensors. Loading is a

conversion pipelinethat transforms them into the runtime layout we want.

The central abstraction introduced by this refactor is dynamic weight loading via a WeightConverter

.

WeightConverter

lets us define:

source key patterns → target key(s) + operations

Primitive operations (chunk, concatenate, etc.) are composable. Two that are particularly useful for MoEs:

MergeModulelist

merges a list of tensors into a single tensor. For example, you can composeMergeModulelist

withConcatenate

to stack the experts in a MoE and pack them into one tensor.WeightConverter( ["block_sparse_moe.experts.*.w1.weight", "block_sparse_moe.experts.*.w3.weight",], "mlp.experts.gate_up_proj", operations=[ MergeModulelist(dim=0), Concatenate(dim=1), ], )

SplitModulelist

splits a tensor back into a list of tensors. For example, you can split a stack of experts back into individual experts.WeightConverter( "mlp.experts.down_proj", "block_sparse_moe.experts.*.w2.weight", operations=[SplitModulelist(dim=0)], )

The refactor improves not just what conversions exist, but how they’re scheduled.

The loader scans checkpoint keys once, matches them against converter patterns, and groups tensors per converter. Once a key is identified as needed, it’s registered as a future and materialized via a thread pool. Conversion operations run only once their dependencies are ready. For example, MergeModulelist

waits until all experts for a layer are loaded.

This avoids repeated scans and reduces memory peaks.

To evaluate the improvements introduced by the new weight-loading pipeline, we benchmarked the v4 vs v5 versions of transformers

. The focus is on loading speed of large MoE models, which is often a bottleneck in training and inference.

We benchmarked v4 vs v5 using:

• v4 branch: https://github.com/ariG23498/transformers/tree/bench-v4
• v5 branch: https://github.com/ariG23498/transformers/tree/bench-v5

Example:

from transformers import AutoModelForCausalLM
model_id = "Qwen/Qwen1.5-110B-Chat"
model = AutoModelForCausalLM.from_pretrained(model_id)

Two relevant environment variables:

HF_ENABLE_PARALLEL_LOADING

: Enables parallel shard loading via threads.HF_DEACTIVATE_ASYNC_LOAD

:Disables the new async pipeline (v5 escape hatch).

Model: Qwen/Qwen1.5-110B-Chat

GPU: 1× A100 (80GB)

| Version | Strategy | Loading Mode | Time | |---|---|---|---| | v4.57.6 | device_map="auto" | Threadpool | 66.24s | | v4.57.6 | device_map="auto" | Sequential | 67.29s | | v4.57.6 | TP | — | OOM | | v5 | device_map="auto" | Async (default) | 20.71s | | v5 | device_map="auto" | Sync | 45.3s | | v5 | TP | Async | 10.1s | | v5 | TP | Sync | 19.28s |