AGENTIC-SYSTEMSJuly 7, 20261 min read

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

As AI accelerates scientific publishing, independent verification has become increasingly difficult. VERITAS introduces a domain-agnostic replication framework powered by coding agents to automate experiment execution, code debugging, and claim verification.

Computer Science > Artificial Intelligence

Title:VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

View PDF HTML (experimental)Abstract:AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

Automated Data Readiness for Scientific AI

Researchers have introduced REDI, an open-source framework that automates the transformation of raw scientific datasets into AI-ready data. Tested on the Frontier supercomputer, REDI addresses the data preparation bottleneck in scientific AI domains.

agentic-systems

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Researchers have introduced ASK+, a novel framework that significantly enhances small language model (SLM) assistance for reinforcement learning agents under partial observability. By providing trajectory-aware context and structured reasoning, ASK+ demonstrates that prompt design and selective gating can outperform sheer model scale.

agentic-systems

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

Researchers have introduced Oyster-II, a reinforcement learning-based constructive safety alignment framework for LLMs. By addressing the limitations of conventional refusal-oriented strategies, Oyster-II achieves an optimal balance between safety and helpfulness, rivaling much larger models.

agentic-systems

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

Researchers introduce SwarmResearch, a novel orchestrator-subagent framework that prevents coding agents from getting stuck in local optima by utilizing a central Shepherd Agent and multiple Search Agents operating on separate git branches.

agentic-systems

Internal Pluralism and the Limits of Pairwise Comparisons

A new study reveals that traditional pairwise comparisons used to align AI with human preferences have critical limitations due to the complexity of human decision-making. By modeling 'internal pluralism', researchers suggest that allowing users to express indecision can significantly improve preference-learning efficiency.

agentic-systems

iFLYTEK-Embodied-Omni Technical Report

iFLYTEK has introduced iFLYTEK-Embodied-Omni, a unified multimodal foundation model that integrates vision, language, and action. By establishing a unique 'brain-cerebellum' collaboration, it addresses the limitations of cascaded pipelines in embodied AI.