让智能体通过状态与策略交替优化,自动把经验变成可复用的能力。
Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
- 用文本状态快速吸收新经验,再逐步提炼为模型参数。
- 在多任务测试中,能力提升显著优于单一优化方式。
- 适合需要持续进化且兼顾响应速度的自主智能体系统。
由大语言模型驱动的自主智能体通过交互不断积累经验,为自我进化提供了可能。核心挑战在于如何将大量任务相关的交互经验转化为可复用的模型能力,同时不牺牲对新证据的快速适应性。显式文本状态(如技能和代理工具)支持快速、可读、可编辑的适应,但依赖外部上下文;参数化策略虽紧凑可重用,更新却较慢。我们提出「经验漏斗」(Experience Funnel),一种耦合快速状态适应与缓慢策略固化的交替循环框架。交互轨迹首先被提炼为显式文本状态,使新经验能快速融入并验证。随后,框架有选择地识别在状态迭代中仍有效的行为,并通过感知转移的蒸馏方式将其固化到策略中。更新后的状态-策略对生成新的轨迹,提供新一轮状态适应与策略固化的数据。跨多个智能体基准的实验表明,该方法在性能上持续优于仅依赖状态或内部化策略的方法,且逐步将有用的显式经验转化为自主策略能力。
原文摘要 · Abstract (English)
Autonomous agents powered by large language models (LLMs) continuously accumulate experience through interaction, creating an opportunity to improve future behavior through self-evolution. A fundamental challenge is how to transform abundant, task-specific interaction experience into reusable model competence without sacrificing the ability to adapt rapidly to newly observed evidence. Explicit textual states, such as skills and agent harnesses, provide fast, human-readable and editable adaptation, but incur persistent dependence on external context; parametric policies provide compact and reusable competence, but are substantially slower to update. We present \textit{Experience Funnel}, a self-evolving framework that couples fast state adaptation with slow policy consolidation in an alternating loop. Interaction trajectories are first distilled into an explicit textual state, where newly acquired experience can be rapidly incorporated and validated. The framework then selectively identifies state-enabled behavior that remains useful across state revisions and consolidates it into the policy through transition-aware distillation. The updated state--policy pair subsequently generates new rollouts, providing fresh evidence for the next round of state adaptation and policy consolidation. Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。