AGENTIC-SYSTEMSJuly 23, 20261 min read

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Researchers introduce LISA, a plug-and-play attention module that reduces computational complexity from O(n^2) to O(nM), achieving a 50% inference speedup and a 5.6% accuracy gain on reasoning benchmarks for long-context models.

Computer Science > Artificial Intelligence

Title:LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

View PDF HTML (experimental)Abstract:Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm. However, the O(n^2) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requires no pretraining from scratch. LISA integrates two lightweight components in parallel within the original model: (1) a Linear Attention module that provides long-range memory with O(n) time complexity; (2) a Lightning Indexer that selects the top-M important tokens from the full context to feed into a Sparse Self-Attention. The two branches are fused via a gating mechanism, reducing inference complexity from O(n^2) to O(nM) (M << n) for generating n tokens. We design a two-stage training pipeline: Stage 1 initializes the model by integrating the linear attention to capture long-range dependencies, complemented by a sliding-window attention mechanism that is optimized via knowledge distillation to approximate the full self-attention distribution of a frozen teacher model. In Stage 2, we further introduce the Indexer to replace the static sliding-window mechanism, enabling dynamic token selection from broader contexts. The Indexer is trained using a novel per-head KL divergence loss, which aligns its selection behavior with the attention patterns of the teacher model. Experiments on DeepSeek-distilled-Qwen models demonstrate that LISA achieves a 50% inference speedup under 16K-token context, while improving average performance by 5.6% on reasoning benchmarks including AIME and MATH-500.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Profile-Graph Memory for LLM Agents: Implicit Cross-Entity Traversal through Narrative Profiles

Researchers introduce ProGraph, a novel two-layer memory architecture designed to enhance multi-hop reasoning and precise recall for LLM agents. Alongside ProGraph, the team released MemHop, a new benchmark tailored for evaluating complex, multi-step memory retrieval.

agentic-systems

NEXUS: Structured Runtime Safety for Tool-Using LLM Agents

NEXUS is a novel structured-plan safety monitor designed for tool-using LLM agents. By combining deterministic rules and calibrated risk scoring, NEXUS enforces graded intervention policies with sub-millisecond latency.

agentic-systems

Information Discernment in Large Language Models

A new study introducing the Learn2Discern (L2D) benchmark reveals that large language models struggle to evaluate source reliability and truthfulness when integrating external knowledge. Researchers found that models often prioritize source popularity over reliability, though simple inference-time interventions can help mitigate these issues.

agentic-systems

OpenEvoShield: Dual Non-Stationary Continual Defense for Open-World Multi-Agent System Attacks

OpenEvoShield is a co-evolutionary continual defense framework designed to protect LLM-based multi-agent systems from dynamic injection attacks in open-world environments by addressing both fast attack evolution and slow normal behavior drift.

agentic-systems

Stochastic Primal-Dual Decoding for Multiobjective Generative Recommender Systems

Researchers introduced a lightweight inference-time decoding layer for generative recommender systems to optimize multi-objective slates without retraining. The stochastic primal-dual approach dynamically balances relevance and constraints, achieving a +1.8% gain in auxiliary objectives in real-world deployment.

agentic-systems

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

A new benchmark evaluates the performance cost of confidential GPU inference on an NVIDIA H100 running under Intel TDX, showing usable throughput despite latency increases.