AGENTIC-SYSTEMSJuly 7, 20261 min read

Automated Data Readiness for Scientific AI

Researchers have introduced REDI, an open-source framework that automates the transformation of raw scientific datasets into AI-ready data. Tested on the Frontier supercomputer, REDI addresses the data preparation bottleneck in scientific AI domains.

Computer Science > Artificial Intelligence

Title:Automated Data Readiness for Scientific AI

View PDF HTML (experimental)Abstract:Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Researchers have introduced ASK+, a novel framework that significantly enhances small language model (SLM) assistance for reinforcement learning agents under partial observability. By providing trajectory-aware context and structured reasoning, ASK+ demonstrates that prompt design and selective gating can outperform sheer model scale.

agentic-systems

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

As AI accelerates scientific publishing, independent verification has become increasingly difficult. VERITAS introduces a domain-agnostic replication framework powered by coding agents to automate experiment execution, code debugging, and claim verification.

agentic-systems

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

Researchers have introduced Oyster-II, a reinforcement learning-based constructive safety alignment framework for LLMs. By addressing the limitations of conventional refusal-oriented strategies, Oyster-II achieves an optimal balance between safety and helpfulness, rivaling much larger models.

agentic-systems

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

Researchers introduce SwarmResearch, a novel orchestrator-subagent framework that prevents coding agents from getting stuck in local optima by utilizing a central Shepherd Agent and multiple Search Agents operating on separate git branches.

agentic-systems

Internal Pluralism and the Limits of Pairwise Comparisons

A new study reveals that traditional pairwise comparisons used to align AI with human preferences have critical limitations due to the complexity of human decision-making. By modeling 'internal pluralism', researchers suggest that allowing users to express indecision can significantly improve preference-learning efficiency.

agentic-systems

iFLYTEK-Embodied-Omni Technical Report

iFLYTEK has introduced iFLYTEK-Embodied-Omni, a unified multimodal foundation model that integrates vision, language, and action. By establishing a unique 'brain-cerebellum' collaboration, it addresses the limitations of cascaded pipelines in embodied AI.