AGENTIC-SYSTEMSDecember 17, 20241 min read20 views

FACTS Grounding: A new benchmark for evaluating the factuality of large language models

Google DeepMind and Google Research have introduced FACTS Grounding, a new benchmark designed to measure the reliability and factuality of large language models (LLMs).

Acknowledgements

FACTS is a collaboration between Google DeepMind and Google Research.

FACTS Grounding was led by: Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Dipanjan Das, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, and Nate Keating.

We are also very grateful for contributions from: Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, and Sasha Goldshtein.

We would also like to thank Avinatan Hassidim, D. Sculley, Fernando Pereira, Koray Kavukcuoglu, Slav Petrov, Ya Xu, and Yossi Matias for their continued support.

Source: Google DeepMind Blog

More in this category

agentic-systems

Align AI to Dynamic Human-AI Workflows

Current AI alignment methods fall short by relying on static representations of human preferences. A new paper proposes a paradigm shift toward "interactive and complementary alignment," allowing AI and humans to co-evolve within dynamic workflows.

agentic-systems

MemoHarness: Agent Harnesses That Learn from Experience

Researchers introduce MemoHarness, an adaptive harness optimization framework that enables AI agents to learn from their own executions. By analyzing performance and storing insights in a dual-layer experience bank, it dynamically adapts to new tasks, outperforming static configurations.

agentic-systems

How Artificial Intelligence LLM Engines Shape the Global Conflict Information Environment

A new study reveals how AI-powered answer engines are highly vulnerable to hallucination and manipulation when reporting on global conflicts. The lack of robust digital records allows malicious actors to exploit these systems through Generative Engine Optimization (GEO), posing a new threat of AI-driven information warfare.

agentic-systems

AI Agents Do Not Fail Alone:The Context Fails First

A new study validates context-engineering quality as an independent leading indicator of AI agent reliability, introducing an open-source evaluation infrastructure called ProofAgent-Harness to predict behavioral outcomes before deployment.

agentic-systems

Interpretable Language Model for Closed-Loop Type 1 Diabetes Control

Researchers have developed LLM-T1D, a novel approach combining reinforcement learning and large language models to automate insulin delivery for Type 1 Diabetes. The system not only outperforms traditional methods but also explains its decisions in plain language, addressing the 'black-box' challenge in medical AI.

agentic-systems

Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models

A new study challenges the assumption that scaling alone guarantees capability convergence in AI models. Instead, researchers demonstrate that true capability depends on 'access structure'—specifically, hybrid architectures combining compressive state channels with scalable verbatim-index channels.