Introspective Diffusion Language Models

Introspective Diffusion

Language Models

vs. LLaDA-2.1-mini 43.3

vs. LLaDA-2.1-mini 30.4

LLaDA-2.1-mini at C=64

to base AR model

Abstract

Diffusion language models (DLMs) offer a compelling promise: parallel token generation could break the sequential bottleneck of autoregressive (AR) decoding. Yet in practice, DLMs consistently lag behind AR models in quality.

We argue that this gap stems from a fundamental failure of introspective consistency: AR models agree with what they generate, whereas DLMs often do not. We introduce the Introspective Diffusion Language Model (I-DLM), which uses introspective strided decoding (ISD) to verify previously generated tokens while advancing new ones in the same forward pass.

Empirically, I-DLM-8B is the first DLM to match the quality of its same-scale AR counterpart, outperforming LLaDA-2.1-mini (16B) by +26 on AIME-24 and +15 on LiveCodeBench-v6 with half the parameters, while delivering 2.9-4.1x throughput at high concurrency. With gated LoRA, ISD enables bit-for-bit lossless acceleration.

Why Introspective Consistency?

Key Insight: AR training unifies generation and introspection in one forward pass. Existing DLMs miss this — they learn to denoise but not to introspect.

We identify three fundamental bottlenecks in current DLMs:

(1) Low introspective consistency. SDAR: 0.699 vs. I-DLM: 0.984.

(2) Compute inefficiency. TiDAR: ~7.8x overhead vs. I-DLM: ~2.5x.

(3) Infrastructure mismatch. SDAR slope=84 vs. I-DLM: 549.

The I-DLM Method

Introspective-Consistency Training

Convert pretrained AR models via causal attention, logit shift, and an all-masked objective.

Introspective Strided Decoding

Generate N tokens per forward pass while verifying prior tokens via the p/q acceptance criterion.

AR-Compatible Serving

Strict causal attention enables direct integration into SGLang with no custom infrastructure.

Decoding paradigm comparison. I-DLM is a drop-in replacement within AR serving infrastructure.

Results

I-DLM is the first DLM to match same-scale AR quality while surpassing all prior DLMs across 15 benchmarks.

End-to-End Quality

Blue = best non-AR <30B. Bold = best non-AR <100B.

| Benchmark | Qwen3 8B | LLaDA-2.1-mini 16B | I-DLM 8B | I-DLM 32B | |---|---|---|---|---| | ARC-C | 95.8 | 90.2 | 95.8 | 96.8 | | MMLU | 83.5 | 74.5 | 82.4 | 86.8 | | GPQA-D | 58.9 | 46.0 | 55.6 | 62.1 | | GSM8K | 96.0 | 89.0 | 95.0 | 94.9 | | MATH-500 | 95.8 | 85.0 | 96.8 | 97.6 | | AIME-24 | 73.1 | 43.3 | 69.6 | 83.3 | | HumanEval | 95.1 | 86.0 | 93.3 | 96.3 | | LCB-v6 | 50.3 | 30.4 | 45.7 | 57.1 |