AGENTIC-SYSTEMSJuly 8, 20261 min read

Akashic: A Low-Overhead LLM Inference Service with MemAttention

Researchers have introduced Akashic, a low-overhead memory system built around MemAttention to address context overload in Large Language Models (LLMs). Akashic improves task accuracy by up to 10.2 points while significantly boosting throughput and system efficiency.

Computer Science > Artificial Intelligence

Title:Akashic: A Low-Overhead LLM Inference Service with MemAttention

View PDF HTML (experimental)Abstract:Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure

Researchers have introduced Onnes, a physics-grounded digital-twin simulator driven by multi-agent LLMs to automate and accurately diagnose faults in dilution refrigerators for quantum computers. By combining real-world physics models with AI, Onnes achieves up to 99% accuracy, paving the way for more stable quantum computing infrastructure operations.

agentic-systems

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

Researchers have introduced FirstResearch, a first-principles framework that makes AI-generated scientific questions auditable through a structured 'Research Question Certificate'.

agentic-systems

ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation

Researchers have introduced ArtisanCAD, an industrial-grade AI agent capable of transforming vague natural language prompts into complete, editable 3D CAD models by leveraging a novel CAD-IR representation and distilling expert knowledge.

agentic-systems

From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space

Current AI agents often rely on passive memory retrieval, limiting their personalization capabilities. The new NapMem framework addresses this by treating long-term memory as a structured action space, enabling agents to actively navigate and inspect memory at multiple granularities.

agentic-systems

Controlling Tool Use with Heading-Specific Activation Steering

Researchers have demonstrated that steering vectors extracted from heading-anchor positions can exert bidirectional causal control over tool-invocation behavior in LLMs, helping suppress unnecessary tool use.

agentic-systems

Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents

A new synthesis paper uncovers critical limitations in LLM agents that are often masked by high benchmark scores. By identifying six core failure clusters, the study reveals that strong sub-task performance does not guarantee end-to-end success in complex, real-world applications.