AGENTIC-SYSTEMSMay 25, 20261 min read10 views

Parallel Context Compaction for Long-Horizon LLM Agent Serving

A new study introduces parallel context compaction to address the inference bottleneck in long-horizon LLM agents, reducing latency and providing fine-grained control over summary volume.

Computer Science > Artificial Intelligence

Title:Parallel Context Compaction for Long-Horizon LLM Agent Serving

View PDF HTML (experimental)Abstract:Long-horizon LLM agents accumulate growing conversation histories that eventually exceed the model's context window. Context compaction via LLM-based summarization keeps the conversation bounded, but summarization is inherently lossy and the blocking call stalls agent inference for tens of seconds. Moreover, the operator has no fine-grained control over summary volume since prompt instructions are largely ignored, and as context grows, both the amount of output tokens the model produces and the information it retains fluctuate substantially from run to run, making the agent's retained knowledge unpredictable across runs. We introduce \textbf{parallel compaction} for long-horizon agentic flows and characterize it against the sequential synchronous baseline across four backbones spanning 8B to 120B parameters, mixing dense and MoE architectures with reasoning and non-reasoning models, on the HotpotQA multi-hop QA and LoCoMo long-context dialogue benchmarks. Parallel compaction gives the operator fine-grained, predictable control over summary volume and enables more targeted prompt engineering per block. At matched compaction decode volume, it reduces end-to-end wall time and improves compaction throughput over the sequential baseline.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models

A new study challenges the assumption that scaling alone guarantees capability convergence in AI models. Instead, researchers demonstrate that true capability depends on 'access structure'—specifically, hybrid architectures combining compressive state channels with scalable verbatim-index channels.

agentic-systems

The Steering Budget: Examples beat Knobs

Generative AI models are traditionally steered using 'knobs' like prompts, but they hit a hard ceiling. A new study reveals that providing concrete examples instead of adjusting prompts unlocks the model's full 'steering budget,' offering far greater reach and expressiveness.

agentic-systems

HG-RAG: Hierarchy-Guided Retrieval-Augmented Generation for Structured Knowledge Graphs

HG-RAG (Hierarchy-Guided Retrieval-Augmented Generation) is a novel framework designed to overcome the limitations of traditional flat RAG systems by traversing hierarchical knowledge graphs, significantly improving multi-hop reasoning and reducing hallucinations in LLMs.

agentic-systems

Enhancing Small Language Models Reasoning through Knowledge Graph Grounding

Researchers have proposed a neuro-symbolic framework that enhances the reasoning capabilities of Small Language Models (SLMs) like Gemma 3 and Llama 3.2 using Knowledge Graphs, achieving up to a 2x performance boost.

agentic-systems

Orchestrating Power Grid Studies with Multi-Agent AI and MCP Servers

This paper explores how Agentic AI and Model Context Protocol (MCP) can support power-grid studies by integrating LLMs with numerical simulation tools under human supervision.

agentic-systems

Human AI Construction of Bayesian Networks for Operational Decision Support -- A Virtual Survey Approach

Researchers propose a new methodology using Large Language Models and AI agents to construct Bayesian Belief Networks, bridging the gap between expert opinion and data-driven learning. The framework is demonstrated through a healthcare decision-making model, revealing key insights into customer behavior.