arXiv:2509.22764cs.LGcs.AI2025-09被引 1

让大模型像人一样持续学习,靠提示词调度实现长期记忆。

In-Context Learning can Perform Continual Learning Like Humans

  • 通过调整提示顺序和任务调度,让模型在不更新参数下持续学习。
  • 特定大模型在分散练习下表现出类似人类的“最佳间隔”记忆效果。
  • 提出人类记忆相似度指标,发现MAMBA等模型更贴近人类遗忘规律。

大型语言模型(LLMs)可通过上下文学习(ICL)在不更新参数的情况下适应新任务,具备快速适应能力。尽管已有大量研究将ICL视为少样本学习方法,但其在任务连续到来时是否具备长期保留与跨任务知识积累仍缺乏探索。受人类记忆研究启发,我们考察了多任务设置下ICL的保留特性,并将其拓展为上下文持续学习(ICCL),通过任务调度与提示重排使持续学习能力自然涌现。在马尔可夫链基准测试中,特定大模型表现出类似人类的分散练习(DP)优势,稳定呈现记忆“甜蜜点”。此外,我们提出人类保留相似性度量,量化持续学习方法与人类保留动态的契合度。结果显示,线性注意力模型如MAMBA和RWKV虽保留性能逊于基于Transformer的模型,但其保留模式更接近人类。总体而言,ICCL兼具认知合理性与实际有效性,提供一种无需参数更新的持续学习范式,缓解灾难性遗忘,解决传统持续学习中的稳定性-可塑性难题。

原文摘要 · Abstract (English)

Large language models (LLMs) can adapt to new tasks via in-context learning (ICL) without parameter updates, making them powerful learning engines for fast adaptation. While extensive research has examined ICL as a few-shot learner, whether it can achieve long-term retention and cross-task knowledge accumulation when multitasks arrive sequentially remains underexplored. Motivated by human memory studies, we investigate the retention characteristics of ICL in multitask settings and extend it to in-context continual learning (ICCL), where continual learning ability emerges through task scheduling and prompt rearrangement. Experiments on Markov-Chain benchmarks demonstrate that, for specific large-language models, ICCL benefits from distributed practice (DP) in a manner analogous to humans, consistently revealing a spacing "sweet spot" for retention. Beyond retention performance, we propose a human-retention similarity metric to quantify how closely a continual-learning (CL) method aligns with human retention dynamics. Using this metric, we show that linear-attention models such as MAMBA and RWKV exhibit particularly human-like retention patterns, despite their retention performance lagging behind that of Transformer-based LLMs. Overall, our results establish ICCL as both cognitively plausible and practically effective, providing an inference-only CL paradigm that mitigates catastrophic forgetting and addresses the stability-plasticity dilemma in conventional CL methods.

持续学习上下文学习记忆机制大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。