arXiv:2511.23436cs.AI2025-11

让智能体自我学习进化,无需人工标注。

Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelliAgent

  • 小模型生成答案,大模型逐步验证,自动构建训练数据。
  • 用双层记忆和重放机制,实现持续积累知识与能力提升。
  • 适合希望打造长期进化智能体的研究者和开发者。

我们提出 SuperIntelliAgent,一个通过自监督交互实现持续智能增长的智能体学习框架。该框架将可训练的小型扩散模型(学习者)与冻结的大语言模型(验证者)结合:学习者生成候选输出,验证者通过逐步推理进行评估,二者交互生成用于直接偏好优化(DPO)的选择/拒绝对,将每个输入转化为伪训练信号以实现持续改进。框架集成双尺度记忆:短期上下文记忆保留迭代优化中的推理轨迹,长期记忆通过轻量级在线微调固化知识。重放缓冲区保存可验证进步的样本并重播作为辅助监督,强化近期学习并形成自适应课程。该框架与基础设施无关,可嵌入现有智能体系统,将普通推理循环转变为终身优化过程。实验表明,仅需少量自动生成的 DPO 对,学习者在所有基准测试中均实现性能提升,证明此机制为持续智能积累与真实部署提供了可行路径。

原文摘要 · Abstract (English)

We introduce SuperIntelliAgent, an agentic learning framework that couples a trainable small diffusion model (the learner) with a frozen large language model (the verifier) to enable continual intelligence growth through self-supervised interaction. Unlike conventional supervised fine-tuning, SuperIntelliAgent learns autonomously without annotation: the learner generates candidate outputs, the verifier evaluates them through step-by-step reasoning, and their interaction produces chosen/rejected pairs for Direct Preference Optimization (DPO). This converts each input into a pseudo-training signal for continual improvement. The framework integrates dual-scale memory: short-term in-context memory that preserves reasoning traces across refinement cycles, and long-term memory that consolidates acquired knowledge through lightweight on-the-fly fine-tuning. A replay buffer retains samples that show verifiable progress and replays them as auxiliary supervision, reinforcing recent learning while forming adaptive curricula. SuperIntelliAgent is infrastructure-agnostic and can be plugged into existing agentic frameworks while turning ordinary inference loops into a lifelong optimization process. We posit that pairing a trainable learner with a reasoning-capable verifier forms a minimal reliable unit of growing intelligence, as paired feedback and partial-history replay yield richer learning curricula and stronger preference alignment. With a small number of automatically generated DPO pairs, the learner improves across all benchmarks, indicating that this mechanism provides a promising direction for continual intelligence accumulation and real-world deployment.

智能体持续学习自我训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。