AGENTIC-SYSTEMSDecember 9, 20251 min read18 views

FACTS Benchmark Suite: Systematically evaluating the factuality of large language models

Google and Kaggle introduce the FACTS Benchmark Suite, a comprehensive toolset designed to measure and improve the factual accuracy of large language models across various scenarios.

Large language models (LLMs) are increasingly becoming a primary source for information delivery across diverse use cases, so it’s important that their responses are factually accurate.

In order to continue improving their performance on this industry-wide challenge, we have to better understand the types of use cases where models struggle to provide an accurate response and better measure factuality performance in those areas.

The FACTS Benchmark Suite

Today, we’re teaming up with Kaggle to introduce the FACTS Benchmark Suite. It extends our previous work developing the FACTS Grounding Benchmark, with three additional factuality benchmarks, including:

A Parametric Benchmark that measures the model’s ability to access its internal knowledge accurately in factoid question use-cases.
A Search Benchmark that tests a model’s ability to use Search as a tool to retrieve information and synthesize it correctly.
A Multimodal Benchmark that tests a model’s ability to answer prompts related to input images in a factually correct manner.

We are also updating the original FACTS grounding benchmark with Grounding Benchmark - v2, an extended benchmark to test a model’s ability to provide answers grounded in the context of a given prompt.

Each benchmark was carefully curated to produce a total of 3,513 examples, which we are making publicly available today. Similar to our previous release, we are following standard industry practice and keeping an evaluation set held-out as a private set. The FACTS Benchmark Suite Score (or FACTS Score) is calculated as the average accuracy of both public and private sets across the four benchmarks. Kaggle will oversee the management of the FACTS Benchmark Suite. This includes owning the private held-out sets, testing the leading LLMs on the benchmarks, and hosting the results on a public leaderboard. More details about the FACTS evaluation methodology can be found in our tech report.

Benchmark overview

Parametric Benchmark

The FACTS Parametric benchmark assesses the ability of models to accurately answer factual questions, without the aid of external tools like web search. All the questions in the benchmark are “trivia style” questions driven by user interest that can be answered via Wikipedia (a standard source for LLM pretraining). The resulting benchmark consists of a 1052-item public set and a 1052-item private set.

Source: Google DeepMind Blog

More in this category

agentic-systems

S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation

Researchers have introduced S1-Omni, a unified multimodal reasoning model designed for AI for Science (AI4S). Outperforming advanced models like GPT-5.5 and Gemini-3.1-Pro on various benchmarks, S1-Omni addresses the fragmentation in modeling complex scientific data.

agentic-systems

NeurOWL: An LLM-Based Neural-symbolic Framework for Incomplete OWL Ontology Reasoning

Researchers have introduced NeurOWL, a novel LLM-based neuro-symbolic framework designed to address reasoning challenges in incomplete OWL ontologies. By unifying subsumption verification and ontology abduction, NeurOWL demonstrates robust performance across multiple domains.

agentic-systems

ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

Researchers introduce ToolVerse, a comprehensive framework designed to scale up agentic reinforcement learning environments and enable LLM agents to perform complex, long-horizon reasoning tasks using thousands of real-world tools.

NOW LET US Related – Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

agentic-systems

Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

A new study introduces a local neuro-symbolic AI pipeline to automate LEED v4.1 green building certification screening. The findings reveal that the small 4-billion-parameter Gemma 3 model outperforms larger models, while incorporating multimodal drawing images consistently degrades performance.

agentic-systems

SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

Researchers have introduced SeerGuard, a proactive safety framework that prevents mobile GUI agents from executing risky actions by predicting consequences beforehand using a Safety-Augmented World Model (SAWM).

NOW LET US Related – MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

agentic-systems

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

Researchers have proposed MGDT, a novel framework for Multimodal Knowledge Graph Completion (MKGC) that utilizes an align-then-diffuse paradigm. By integrating a frozen MLLM and a Relation-Adaptive Mixture-of-Experts module, MGDT significantly outperforms existing baselines.