AGENTIC-SYSTEMSJuly 28, 20261 min read

MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

Researchers have proposed MIITA, a memory-induced inference-time adaptation framework enabling small language models to achieve continual learning without catastrophic forgetting under strict storage constraints.

Computer Science > Artificial Intelligence

Title:MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

View PDF HTML (experimental)Abstract:Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments. However, directly updating their limited parameter space causes catastrophic forgetting. While memory-based methods naturally address this by decoupling knowledge retention from parameters, existing approaches designed for large language models (LLMs) rely on abundant storage and strong in-context reasoning that SLMs lack. To address these challenges, we propose MIITA, a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage. MIITA stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues. The retrieved directions are applied through gated temporary hidden-state adaptation, enabling non-destructive reuse of past supervision without backbone updates, prompt extensions, or test-time backpropagation. A local theoretical analysis links this design to first-order loss reduction, uncertainty-guided retrieval, and directional coverage for retaining old-stage knowledge. Extensive experiments across diverse supervised CL settings show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Keyword Matters: Unveiling the Energy Sensitivity of On-Device LLM Prompting

A new empirical study reveals that prompt wording and instruction structure significantly impact the energy consumption of on-device LLMs. Researchers highlight prompt engineering as a lightweight yet effective lever for improving battery efficiency on mobile devices.

agentic-systems

Reference Feature Atlases for Mechanistic Auditing of Language Models

Researchers propose 'Reference Feature Atlases,' a novel framework that enables efficient mechanistic auditing of large language models without retraining internal features from scratch. By leveraging reference feature libraries and residual channels, the method effectively detects hidden objectives and subtle behavioral biases across different LLMs.

agentic-systems

Concept-based Visual Counterfactual Explanations with Diffusion Models

Researchers introduced C-VCE, a novel diffusion framework that generates human-interpretable visual counterfactual explanations by building concept bottleneck layers directly into the generative model.

agentic-systems

Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents

Researchers have developed ScenarioGeneratorAgent, an automated pipeline that generates synthetic evaluation scenarios for Industry 4.0 AI agents. Tested on Smart Grid Transformers, the approach accelerates benchmark generation by 8x while maintaining high quality and adherence to IEC standards.

agentic-systems

Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

A new study reveals that large language models (LLMs) often produce inconsistent answers when the same question is rephrased, with mismatch rates exceeding 23%. The findings suggest that standard benchmark accuracy masks substantial instability, but a self-paraphrasing strategy can help recover latent knowledge at inference time.

agentic-systems

DeepLens Diagnosis Agent: Agentic Workflow Design Lets a Small Reasoning Model Compete with Frontier LLMs

By leveraging a structured five-stage agentic workflow, the 7-billion-parameter DeepLens Diagnosis Agent outperforms frontier models like Claude Sonnet 4.5 and Gemini 3.1 Pro on complex clinical diagnostics, demonstrating that disciplined process design can outweigh raw parameter count and API costs.