AGENTIC-SYSTEMSJuly 17, 20261 min read

Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation

Researchers propose an automated, agentic red-teaming framework that systematically synthesizes difficult adversarial examples to improve the safety of Multimodal Large Language Models (MLLMs). By leveraging a multi-agent architecture, the system reduces the False Negative Rate in image safety benchmarks from 41.2% to 24.5% without human intervention.

Computer Science > Artificial Intelligence

Title:Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation

View PDF HTML (experimental)Abstract:Multimodal Large Language Models (MLLMs) are increasingly deployed for nuanced content safety and moderation tasks, yet they remain vulnerable to adversarial attacks and out-of-distribution edge cases. Traditional active learning and manual annotation fail to scale against the complexity and volume of novel multimodal threats. In this paper, we propose an automated, agentic red-teaming framework that systematically synthesizes difficult examples using an iterative strategy that proposes novel hypotheses as well as mutating on past attempts. Leveraging a multi-agent architecture that consists of a high-reasoning Architect agent, an advanced image generator, and a multi-level verification committee of LLM raters, our system autonomously uncovers boundary-pushing violations and ambiguous policy edge cases without any human intervention. By employing these carefully synthesized adversarial examples as in-context demonstrations via test-time Retrieval, we substantially improve the target model's robustness, reducing the False Negative Rate (FNR) from 41.2% to 24.5% in a public image safety benchmark without relying on any human labeling.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

RegNetAgents: A Multi-Agent Framework for Cross-Network Regulatory Driver Identification in Cancer Genomics

Researchers have introduced RegNetAgents, an AI-oriented multi-agent framework designed to identify regulatory driver candidates across heterogeneous gene networks in cancer genomics. By integrating bulk tumor and single-cell data, the system offers a powerful tool for discovering therapeutic targets in breast and colorectal cancers.

agentic-systems

IMEX Interaction-Based Model Explanation

The IMEX (Interaction-Based Model Explanation) framework offers a novel approach to explainable AI by identifying key features and complex, higher-order interactions within black-box predictive models.

agentic-systems

DialogueVPR: Towards Conversational Visual Place Recognition

Researchers have introduced DialogueVPR, a novel paradigm that casts visual place recognition as an interactive, dialogue-driven reasoning process. By coupling a cross-modal retriever with an intelligent questioner called DQ-pilot, this framework effectively resolves ambiguous real-world descriptions and significantly outperforms traditional static retrieval methods.

agentic-systems

Intelligent Three Level Learning Architecture for Autonomous UAV Swarms in Search and Rescue

Researchers have proposed a novel three-level hierarchical learning architecture for autonomous UAV swarms in search and rescue operations, mimicking biological reflexes, skills, and reasoning.

agentic-systems

ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability

Researchers introduce ToolAnchor, a novel framework that helps LLM agents seamlessly adapt to new tools by injecting counterfactual anchor contexts, effectively overcoming the challenge of behavioral inertia.

agentic-systems

When a Verified World Model Still Loses: Play-Adequacy vs Prediction-Accuracy in LLM-Synthesized Code World Models

A new study reveals that LLM-synthesized Code World Models can achieve near-perfect prediction accuracy yet fail systematically in actual play, exposing a critical 'verified-vs-correct' gap in planning-oriented AI.