AGENTIC-SYSTEMSJuly 28, 20261 min read

Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

A new study reveals that large language models (LLMs) often produce inconsistent answers when the same question is rephrased, with mismatch rates exceeding 23%. The findings suggest that standard benchmark accuracy masks substantial instability, but a self-paraphrasing strategy can help recover latent knowledge at inference time.

Computer Science > Artificial Intelligence

Title:Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

View PDF HTML (experimental)Abstract:Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways. In this work, we study how model answers change under meaning-preserving paraphrases across factual question answering and mathematical reasoning tasks. Across four benchmarks and 13 models, we find that model outputs frequently depend on the exact wording of the prompt. While overall accuracy typically changes only modestly across paraphrases, instance-level behavior is far less stable: for many questions, models alternate between correct and incorrect answers depending on phrasing, with mismatch rates reaching more than 23%. Conditioning on questions that are answered correctly in their original form reveals even larger failures measured by answer flip rates, showing that single-prompt correctness is often a poor indicator of reliability. At the same time, we find that models often produce a correct answer for at least one paraphrase of a question, suggesting that the underlying knowledge is present but inconsistently retrieved. Building on this observation, we show that a simple self-paraphrasing strategy can partially recover this latent knowledge and improve performance at inference time. Together, these findings suggest that standard accuracy metrics can mask substantial instability, and that evaluating consistency across equivalent inputs provides a clearer picture of LLM reliability.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Concept-based Visual Counterfactual Explanations with Diffusion Models

Researchers introduced C-VCE, a novel diffusion framework that generates human-interpretable visual counterfactual explanations by building concept bottleneck layers directly into the generative model.

agentic-systems

Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents

Researchers have developed ScenarioGeneratorAgent, an automated pipeline that generates synthetic evaluation scenarios for Industry 4.0 AI agents. Tested on Smart Grid Transformers, the approach accelerates benchmark generation by 8x while maintaining high quality and adherence to IEC standards.

agentic-systems

DeepLens Diagnosis Agent: Agentic Workflow Design Lets a Small Reasoning Model Compete with Frontier LLMs

By leveraging a structured five-stage agentic workflow, the 7-billion-parameter DeepLens Diagnosis Agent outperforms frontier models like Claude Sonnet 4.5 and Gemini 3.1 Pro on complex clinical diagnostics, demonstrating that disciplined process design can outweigh raw parameter count and API costs.

agentic-systems

MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Models

MedLoCoMo is a new medical benchmark designed to evaluate LLMs' clinical reasoning over multi-session, long-context patient histories. Experiments reveal that reasoning across different hospital admissions remains particularly difficult for current AI models.

agentic-systems

SCAIR: Schema-Conditioned Agentic Iterative Reasoning for Enterprise Knowledge Graphs

Researchers have introduced SCAIR, a training-free agentic framework that enhances Knowledge Graph-based RAG (KG-RAG) for enterprise applications by injecting schema constraints into multi-hop iterative reasoning.

agentic-systems

Loss-Aware Feature-Map Pruning in Convolutional Neural Networks Using Multi-Armed Bandits

Researchers have proposed a loss-aware structured feature-map pruning framework for convolutional neural networks using multi-armed bandits. The approach significantly reduces computational complexity and storage costs while maintaining accuracy levels close to original unpruned models.