AGENTIC-SYSTEMSJuly 24, 20261 min read

Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

NOW LET US Article – Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

A new study reveals that stochastic sampling via temperature variation in a single LLM only provides per-question uncertainty, failing to capture complex cross-question epistemic structures compared to a diverse ensemble of distinct models.

Computer Science > Artificial Intelligence

Title:Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

View PDF HTML (experimental)Abstract:When a language model gives different answers on repeated runs, does that variation reveal what it does not know? Self-consistency turns the variation into a per-question uncertainty estimate via majority voting. But does the same variation reveal cross-question structure -- related questions flipping together, the way a diverse ensemble does? We compare two regimes on the same questions: one model run $100$ times at $\tau=1$ versus an ensemble of $24$ LLMs run once each at $\tau=0$. A Marchenko--Pastur random-matrix test separates signal from sampling noise on both sides. Within any single model, at most one dimension rises above noise across five families and three benchmarks (MMLU, HellaSwag, GSM8K). Across the ensemble, four eigenvalues clear the noise edge, while a matched-difficulty Bernoulli null produces at most one in $500$ Monte Carlo draws. Self-consistency gives accurate per-question uncertainty but no detectable cross-question structure; only a diverse ensemble surfaces what a model does not know.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

InferenceBench is a new benchmark evaluating AI agents on open-ended LLM inference optimization tasks under realistic constraints. While agents outperform default setups, qualitative analysis reveals key limitations in their strategic exploration capabilities.

agentic-systems

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

Researchers have introduced PersonaTrail, a novel benchmark evaluating personalized web agents through realistic browsing histories. Alongside it, the PACMem memory framework structures raw browsing data to significantly boost agent performance in inferring user preferences.

agentic-systems

Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs

A new study reveals that deception detection in LLMs heavily depends on lie typology and training data structure, highlighting why current detection probes struggle in out-of-domain scenarios.

agentic-systems

Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

A new benchmark evaluates five leading large language models on multi-sensor physical hazard assessments, revealing a critical vulnerability when evaluating combined sensor risks below individual thresholds. Despite near-perfect accuracy on single-sensor breaches, all models failed to issue precautionary signals for multi-sensor hazards.

agentic-systems

DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decoding

Researchers have introduced DC-Leap, a training-free framework that accelerates inference in Diffusion Large Language Models (dLLMs) by up to 105x when combined with KV-Cache while maintaining generation quality.

agentic-systems

PlanE: Meta Planning of Data, Tuning, and Inference for Extractive-based LLMs

Researchers introduce PlanE, a novel framework featuring a Data-Tuning-Inference (DTI) planner designed to optimize data, tuning, and inference processes for extractive-based LLMs while reducing annotation costs.