AGENTIC-SYSTEMSMay 6, 20261 min read5 views

Iterative Finetuning is Mostly Idempotent

Researchers found that training AI models on their own outputs (iterative finetuning) typically leads to behavioral decay or stability rather than amplification of biases, suggesting a natural limit to how much negative traits can escalate.

Computer Science > Artificial Intelligence

Title: Iterative Finetuning is Mostly Idempotent

Abstract: If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation

Researchers have introduced S1-Omni, a unified multimodal reasoning model designed for AI for Science (AI4S). Outperforming advanced models like GPT-5.5 and Gemini-3.1-Pro on various benchmarks, S1-Omni addresses the fragmentation in modeling complex scientific data.

agentic-systems

NeurOWL: An LLM-Based Neural-symbolic Framework for Incomplete OWL Ontology Reasoning

Researchers have introduced NeurOWL, a novel LLM-based neuro-symbolic framework designed to address reasoning challenges in incomplete OWL ontologies. By unifying subsumption verification and ontology abduction, NeurOWL demonstrates robust performance across multiple domains.

agentic-systems

ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

Researchers introduce ToolVerse, a comprehensive framework designed to scale up agentic reinforcement learning environments and enable LLM agents to perform complex, long-horizon reasoning tasks using thousands of real-world tools.

NOW LET US Related – Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

agentic-systems

Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

A new study introduces a local neuro-symbolic AI pipeline to automate LEED v4.1 green building certification screening. The findings reveal that the small 4-billion-parameter Gemma 3 model outperforms larger models, while incorporating multimodal drawing images consistently degrades performance.

agentic-systems

SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

Researchers have introduced SeerGuard, a proactive safety framework that prevents mobile GUI agents from executing risky actions by predicting consequences beforehand using a Safety-Augmented World Model (SAWM).

NOW LET US Related – MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

agentic-systems

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

Researchers have proposed MGDT, a novel framework for Multimodal Knowledge Graph Completion (MKGC) that utilizes an align-then-diffuse paradigm. By integrating a frozen MLLM and a Relation-Adaptive Mixture-of-Experts module, MGDT significantly outperforms existing baselines.