AGENTIC-SYSTEMSJuly 13, 20261 min read1 views

KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

Researchers have introduced KV-PRM, a highly efficient process reward model that addresses the computational bottleneck in multi-agent LLM systems. By directly utilizing the KV cache, KV-PRM reduces scoring FLOPs by up to 5,000x and latency by 37x compared to traditional text-based PRMs.

Computer Science > Artificial Intelligence

Title:KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

View PDF HTML (experimental)Abstract:Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L^2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

NOW LET US Related – YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate

agentic-systems

YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate

Current LLM-based optimization pipelines are fragile due to their reliance on single-point assumptions. YUKTI addresses this by introducing an uncertainty-typed proposition graph and Assumption-Robust Pareto Frontiers, reducing decision regret by over 90% compared to naive approaches.

agentic-systems

From ML Predictions to Informed Diagnostic Assistance Using the Toulmin Model of Argumentation

Researchers have proposed a new approach based on the Toulmin model of argumentation to structure and interpret AI-based image diagnoses. By combining specialized models like MedGemma and MedSigLip, the framework allows medical experts to critically assess ML-generated diagnoses rather than accepting them at face value.

agentic-systems

Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

A new study reveals that minor formatting changes in prompt wrappers can significantly alter LLM performance scores, potentially flipping leaderboard rankings. Researchers introduce two new metrics, FSI and PSI, to measure this sensitivity and offer crucial recommendations for benchmarking and structured-output deployments.

agentic-systems

Interpreting Latent CoT Reasoning as Dynamical Systems

Researchers have introduced a breakthrough framework to decode the "black box" of latent Chain-of-Thought (CoT) reasoning in AI. By applying dynamical systems analysis, the study reveals how models like CODI and COCONUT evolve their reasoning trajectories within the representation space.

agentic-systems

Communication-Efficient Digital-Twin Coordination for Heterogeneous LLM Embodied Agents over Computing Power Networks

Researchers have proposed LDT-Coord, a lightweight digital twin-based coordination framework for heterogeneous LLM-powered embodied agents. The solution addresses network congestion and latency issues by reducing communication overhead by over 70x while maintaining optimal task performance.

agentic-systems

GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning

Researchers have developed GATS, a breakthrough planning framework for AI agents that completely eliminates LLM calls during inference. Achieving a 100% success rate in complex tasks, GATS paves the way for highly efficient and deterministic autonomous agents.