arXiv:2606.04703cs.CLcs.LG2026-06被引 1

让大模型持续学习更稳定,关键在抽象经验、分步注入和高质量训练。

Rethinking Continual Experience Internalization for Self-Evolving LLM Agents

论文配图:Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
图 1 · 摘自论文原文
  • 用抽象原则替代具体实例,提升经验可复用性。
  • 分步注入经验比一次性注入效果更好,尤其适合长流程任务。
  • 用高质量教师轨迹离线训练,避免错误累积导致性能下降。

经验内化将过往交互的上下文经验转化为可重用的参数化能力,为大语言模型的持续学习提供了可行路径。现有方法多关注单轮迁移,我们发现多轮学习中,现有方法反而出现能力逐步退化而非持续提升。通过三个维度系统分析:(1) 经验粒度:原则级经验比实例级更持久,能有效剥离特定轨迹细节,提取可迁移策略;(2) 经验注入方式:分步注入优于全局注入,因其与中间决策状态对齐,对长程工具使用至关重要;(3) 内化机制:基于高质量教师轨迹的离线策略上下文蒸馏,比在线策略蒸馏更稳定,后者受限于学生自身错误状态带来的局部修正。这些发现共同形成一套简单但稳健的经验内化方案,为构建自演化、持续学习的大模型智能体提供明确工程指导。

原文摘要 · Abstract (English)

Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.

持续学习经验内化大模型智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。