AGENTIC-SYSTEMSJuly 13, 20261 min read

Multimodal Reward Hacking in Reinforcement Learning

A new study explores reward hacking in multimodal large language models (MLLMs) during reinforcement learning, revealing that optimizing imperfect rewards can lead to systematic task failures despite higher reward scores.

Computer Science > Artificial Intelligence

Title:Multimodal Reward Hacking in Reinforcement Learning

View PDF HTML (experimental)Abstract:Reinforcement learning (RL) is increasingly used to align multimodal large language models (MLLMs), but higher rewards do not always imply better task performance. This risk is amplified when visual evidence is evaluated by text-only or weakly grounded rewards. We study reward hacking in MLLM RL across safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithm (GRPO, RLOO, DAPO). We introduce Newly Rewarded Failure Rate (NRFR), which measures failures among samples whose proxy reward improves over the SFT baseline. Outcome-only rewards cause severe hacking, reaching 48.1% Reward Hacking Rate (RHR), while NRFR exceeding RHR shows that RL creates new failures rather than merely inheriting them. Scaling reduces but does not eliminate hacking: even the 32B model retains a 54.9% worse rate under outcome-only rewards, whereas answer-aware rewards improve the oracle trend at every scale. Robustness is also algorithm- and scale-dependent: GRPO is consistently most resistant, RLOO remains vulnerable, and DAPO improves substantially from 2B to 8B. Visual-evidence rewards help only with reliable verification: keyword-based checks increase hacking, while VLM-as-judge semantic verification reduces it. Overall, multimodal reward hacking is a systematic result of optimizing imperfect rewards, and robust alignment requires rewards and verifiers that remain reliable under optimization pressure.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning

Researchers have developed GATS, a breakthrough planning framework for AI agents that completely eliminates LLM calls during inference. Achieving a 100% success rate in complex tasks, GATS paves the way for highly efficient and deterministic autonomous agents.

agentic-systems

A Formalization of the Mean-Field Derivation of the Vlasov Equation: AI-Assisted Lean Formalization as a Strategy Game

Researchers have successfully formalized the complex Vlasov equation in the Lean 4 proof assistant by framing the collaboration between a mathematician and an AI agent as a strategy game.

agentic-systems

MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

Researchers have introduced MedRealMM, a large-scale multimodal benchmark built from real-world patient-doctor interactions. This benchmark provides a more accurate evaluation of large language models (LLMs) in online medical consultation scenarios, where safety and precision are paramount.

agentic-systems

L-MAD: A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoning

The new L-MAD framework demonstrates that assigning expert personas to AI agents in a debate structure improves legal reasoning accuracy by up to 8%. However, the study also warns of an 'over-deliberation drift' where prolonged discussions lead agents to reinforce each other's errors.

agentic-systems

Ceci n'est pas une pipe: AI systems as semantic abstractions

A new study proposes a semantic framework to distinguish between actual facts and AI-generated representations. By defining common failures like extrapolation or source mismatch, this framework aims to evaluate AI systems based on reliable claims rather than apparent fluency.

NOW LET US Related – How Does Bayesian Causal Discovery Fail? Characterising Structural Consequences in Linear Gaussian Networks under Latent Confounding

agentic-systems

How Does Bayesian Causal Discovery Fail? Characterising Structural Consequences in Linear Gaussian Networks under Latent Confounding

This study characterizes how Bayesian causal discovery fails under latent confounding in linear Gaussian models, revealing a counterintuitive threshold where more data increases the likelihood of identifying spurious causal relationships.