BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

Researchers have introduced BV-Blend, a critic-free reinforcement learning framework that stabilizes advantage estimation in LLM alignment. By blending historical data with local statistics, BV-Blend overcomes a critical limitation of GRPO-style methods during cold-start training with binary rewards.

Computer Science > Artificial Intelligence

Title:BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

View PDF HTML (experimental)Abstract:Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models. However, GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable. In particular, when all rollouts in a prompt group receive identical rewards, the within-group reward variance becomes zero, and group normalization yields zero advantages for that group, impeding learning in cold-start regimes with binary verifiers. We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. BV-Blend maintains EMA-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean (SEM) proxy, and uses this weight to blend historical and prompt-local baseline and variance statistics into a standardized advantage for PPO-style clipped updates. Experiments on verifiable reasoning benchmarks show that BV-Blend improves training stability and performance, and remains robust in regimes where group-normalized methods may stall.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Data and Evaluation Closed-Loop for Model Capability Enhancement

A new study introduces the 'capability slice' to establish a closed loop between evaluation and pre-training data for LLMs. This systematic approach allows engineers to pinpoint model weaknesses and perform targeted data interventions instead of relying on intuition.

agentic-systems

Search for Truth from Reasoning: A Dynamic Representation Editing Framework for Steering LLM Trajectories

Researchers have introduced DynaSteer, a dynamic representation editing framework designed to guide Large Language Models (LLMs) toward truthful reasoning trajectories. By dynamically intervening at high-entropy decision points, DynaSteer improves reasoning accuracy without disrupting correct logical paths.

agentic-systems

Primary ICD Category Prediction using LLM-based Probing

Researchers evaluated whether frozen medical large language model (LLM) representations can serve as a shared embedding space for multimodal primary diagnosis category prediction, achieving high accuracy and efficient cross-dataset transfer.

agentic-systems

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

Researchers have proposed ComMem, an innovative test-time adaptation (TTA) approach for vision-language models that mimics the complementary memory systems of the biological brain to enhance practical adaptability.

agentic-systems

TrajRS: Towards Certified Robustness in Pedestrian Trajectory Prediction

Researchers have introduced TrajRS, a new framework that enhances and certifies the robustness of pedestrian trajectory prediction models against adversarial attacks, marking a significant step toward safer autonomous driving systems.

agentic-systems

An AI agent for treatment reasoning over a biomedical tool universe

Researchers have introduced ATHENA-R1, a breakthrough AI agent capable of advanced treatment reasoning across all FDA-approved drugs since 1939. Trained via reinforcement learning over 212 biomedical tools, it outperforms leading models like GPT-5 in complex medical decision-making.

EXPLORE TOPICS