AGENTIC-SYSTEMSApril 3, 20261 min read11 views

Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Models

Researchers propose Truth AnChoring (TAC), a post-hoc calibration method to improve uncertainty estimation in LLMs by aligning model scores with factual correctness. This approach effectively addresses 'proxy failure' and enhances the reliability of AI outputs even with limited supervision.

Computer Science > Artificial Intelligence

Title: Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Models

Uncertainty estimation (UE) aims to detect hallucinated outputs of large language models (LLMs) to improve their reliability. However, UE metrics often exhibit unstable performance across configurations, which significantly limits their applicability. In this work, we formalise this phenomenon as proxy failure, since most UE metrics originate from model behaviour, rather than being explicitly grounded in the factual correctness of LLM outputs. With this, we show that UE metrics become non-discriminative precisely in low-information regimes. To alleviate this, we propose Truth AnChoring (TAC), a post-hoc calibration method to remedy UE metrics, by mapping the raw scores to truth-aligned scores. Even with noisy and few-shot supervision, our TAC can support the learning of well-calibrated uncertainty estimates, and presents a practical calibration protocol. Our findings highlight the limitations of treating heuristic UE metrics as direct indicators of truth uncertainty, and position our TAC as a necessary step toward more reliable uncertainty estimation for LLMs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification

Researchers have introduced VeriSimpl, a novel framework that leverages LLMs and simplification-based verification to accurately translate natural language descriptions into executable optimization formulations.

agentic-systems

SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification

Researchers introduced SonicSampler, a unified suite of tile-aware Triton kernels designed to accelerate LLM sampling and speculative verification. By fusing the entire pipeline into a single CUDA Graph-compatible kernel, SonicSampler achieves up to 16x speedups over state-of-the-art baselines.

agentic-systems

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

Researchers have introduced PersonaTrail, a novel benchmark evaluating personalized web agents through realistic browsing histories. Alongside it, the PACMem memory framework structures raw browsing data to significantly boost agent performance in inferring user preferences.

agentic-systems

Benchmarking the Personalization Capabilities of Large Language Models

A new study introduces SDR-Bench and SDR-Arena to benchmark the personalization capabilities of LLMs in two-party persuasion scenarios, revealing a personalization plateau among frontier models.

agentic-systems

Incomplete Prompt Jailbreaks in Large Language Models

Researchers have conceptualized Incomplete Prompt Jailbreaks (IPJ), a vulnerability where incomplete harmful prompts bypass LLM safeguards due to delayed refusal behaviors. The discovery of functional neurons offers a novel path for precise neuron-level defense interventions.

agentic-systems

Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

A new benchmark evaluates five leading large language models on multi-sensor physical hazard assessments, revealing a critical vulnerability when evaluating combined sensor risks below individual thresholds. Despite near-perfect accuracy on single-sensor breaches, all models failed to issue precautionary signals for multi-sensor hazards.