小模型持续学习新知识不遗忘,靠记忆检索修正推理过程。
MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

- 用语义锚点存经验,推理时按需调用修正方向。
- 固定内存下性能提升,遗忘率显著降低。
- 适合资源受限场景的小模型持续学习应用。
持续学习对资源受限环境中的小型语言模型适应现实需求至关重要。直接更新其有限参数易导致灾难性遗忘。现有基于记忆的方法虽能解耦知识保留与参数更新,但多为大模型设计,依赖充足存储和强上下文推理能力,小模型难以满足。为此,我们提出MIITA:一种在有限存储下针对监督式持续学习的推理时自适应框架。该方法将监督经验以紧凑的修正方向原型形式存储,并利用语义与不确定性线索在推理时检索。检索到的方向通过门控机制临时调整隐藏状态,实现非破坏性复用历史知识,无需更新主干网络、扩展提示或测试时反向传播。局部理论分析表明,该设计可实现一阶损失下降、不确定性引导检索及方向覆盖以保留旧知识。大量实验表明,MIITA在多种监督式持续学习设置下均能稳定提升最终性能并缓解遗忘,在固定内存预算下表现优异。
原文摘要 · Abstract (English)
Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments. However, directly updating their limited parameter space causes catastrophic forgetting. While memory-based methods naturally address this by decoupling knowledge retention from parameters, existing approaches designed for large language models (LLMs) rely on abundant storage and strong in-context reasoning that SLMs lack. To address these challenges, we propose MIITA, a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage. MIITA stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues. The retrieved directions are applied through gated temporary hidden-state adaptation, enabling non-destructive reuse of past supervision without backbone updates, prompt extensions, or test-time backpropagation. A local theoretical analysis links this design to first-order loss reduction, uncertainty-guided retrieval, and directional coverage for retaining old-stage knowledge. Extensive experiments across diverse supervised CL settings show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。