arXiv:2605.16857cs.AI2026-05

让智能体学会动态构建记忆,提升多模态任务表现。

Learning to Learn from Multimodal Experience

论文配图:Learning to Learn from Multimodal Experience
图 1 · 摘自论文原文
  • 用可学习的机制动态设计记忆结构,而非固定模板。
  • 在多模态任务中显著提升性能与泛化能力。
  • 适合需要持续学习与复杂感知的任务场景。

基于经验的学习已成为智能体通过交互轨迹积累并复用过往经验以提升能力的有前景范式。然而,现有方法主要面向文本场景,依赖人工设计的记忆架构,难以适配多模态环境。现实世界中的经验天然具有多模态性,涵盖感知、推理和动作等异构信号,使有效记忆设计更具挑战性。尤其,最优记忆结构高度依赖任务且随时间演变,固定设计无法满足需求。本文提出「从多模态经验中学习如何学习」的新范式,将记忆设计从预定义组件转变为可适应、可学习的过程。所提框架使智能体能根据任务需求与交互历史动态构建、组织和利用记忆,有效学习如何组织经验以提升性能。实验表明,自适应记忆设计显著增强智能体在多模态任务上的表现与泛化能力,凸显学习记忆机制在基于经验学习中的关键作用。

原文摘要 · Abstract (English)

Experience-driven learning has emerged as a promising paradigm for enabling agents to improve from interaction trajectories by accumulating and reusing past experience. However, existing approaches are predominantly developed in textual settings and rely on manually designed memory schemas, limiting their applicability to multimodal environments. In real-world scenarios, experience is inherently multimodal, involving heterogeneous signals across perception, reasoning, and action, which makes effective memory design significantly more challenging. In particular, the optimal way to structure and utilize multimodal experience is highly task-dependent and evolves over time, rendering fixed memory designs insufficient. In this work, we propose a new paradigm, learning to learn from multimodal experience, which shifts memory design from a predefined component to an adaptive and learnable process. Our framework enables agents to dynamically construct, organize, and utilize memory based on task requirements and interaction history, effectively learning how to structure experience for improved performance. Experiments demonstrate that adaptive memory design substantially enhances agent performance and generalization across multimodal tasks, highlighting the critical role of learning memory mechanisms in experience-driven learning.

多模态持续学习记忆机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。