AGENTIC-SYSTEMSApril 21, 20261 min read7 views

Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals

This paper explains why training-free token reduction methods suffer from a 'cliff-like' performance collapse at high compression rates and introduces CATIS, a method using stable unary signals to maintain accuracy.

Computer Science > Artificial Intelligence

Title: Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals

Training-free token reduction methods for Vision Transformers (ToMe, ToFu, PiToMe, and MCTF) employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression. This paper explains why. We develop a diagnostic framework with two tools, ranking consistency $\rho_s$ and off-diagonal correlation $\rho_\text{off}$, that decomposes the collapse into (1) a signal-agnostic error amplifier inherent to layer-wise reduction, predicting convex Pareto curves and $r_{\text{crit}} \propto 1/L$; and (2) shared reliance on pairwise similarity signals whose ranking consistency degrades from $\rho_s=0.88$ to $0.27$ in deep layers. Pairwise rankings are inherently unstable ($O(N_p^2)$ joint perturbations) while unary signals enjoy greater stability ($O(N_p)$ perturbations, CLT). From three design principles derived from this diagnosis, we construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain. On ViT-Large at 63% FLOPs reduction, CATIS retains 96.9% of vanilla accuracy (81.0%) on ImageNet-1K where all baselines collapse to 43--65%.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

Researchers demonstrate that Masked Diffusion Language Models (MDLMs) serve as highly effective, steerable text-based world models for agentic reinforcement learning. By leveraging bidirectional denoising, MDLMs outperform autoregressive models four times their size in coherence, groundedness, and rollout diversity.

NOW LET US Related – Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment

agentic-systems

Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment

A new study demonstrates that small language models (SLMs) under 3 billion parameters can serve as highly capable local experts for specialized tasks. By combining structured benchmarking with low-cost parameter-efficient fine-tuning (PEFT), institutions can achieve AI autonomy without relying on expensive hardware.

agentic-systems

PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injection

Researchers have introduced PlanFlip, a novel prompt injection attack framework targeting the planning phase of multi-agent LLM systems. The study reveals critical security blind spots in homogeneous agent pipelines and demonstrates that reasoning-augmented models like DeepSeek-R1 exhibit strong resistance.

agentic-systems

Rater State Bias in RLHF Preference Data: An Audit Framework

A new study identifies a structured bias in Reinforcement Learning from Human Feedback (RLHF) caused by the psychological state of human raters. Under stress, raters' shifting preferences can propagate through reward modeling, potentially compromising AI policy optimization.

agentic-systems

A Survey on GNN-based Link Prediction: Techniques, Applications, and Challenges

A comprehensive survey paper provides a systematic review of GNN-based link prediction, categorizing key encoder architectures and real-world applications while highlighting future research directions.

agentic-systems

Some Large Language Models Exhibit Consistent Risk Attitudes

A new study reveals that some large language models (LLMs) exhibit systematic and consistent risk attitudes across different domains, such as finance and medicine. This discovery provides a new foundation for evaluating and aligning AI behaviors in high-stakes decision-making environments.