arXiv:2605.31075cs.CV2026-05

让多模态智能体学会只记对任务有用的信息,提升长期记忆效率。

Task-Focused Memorization for Multimodal Agents

论文配图:Task-Focused Memorization for Multimodal Agents
图 1 · 摘自论文原文
  • 用强化学习训练记忆策略,动态决定该记住什么内容。
  • 在三个视频基准上,问答准确率分别提升5.3%至7.0%。
  • 适合需要持续学习的智能体场景,如机器人或虚拟助手。

长期记忆对多模态智能体构建连贯经验、积累世界知识和实现持续学习至关重要。然而,有效构建记忆不仅依赖模块设计与准确性要求,更核心的挑战在于判断该记住什么。多模态智能体如具身智能体,在真实或虚拟环境中持续感知、推理与行动,接收无限的多模态观测流。从这一信息爆炸中,智能体必须有选择性地保留对其角色相关且未来任务有价值的内容。为此,我们将记忆生成建模为可学习的记忆策略,提出基于强化学习的TaskMem(任务聚焦记忆策略学习)框架,使策略能根据环境中的实际任务动态调整关注点。TaskMem采用两阶段训练:第一阶段优化记忆质量,满足基本保真度要求;第二阶段部署后,通过在基础多模态大模型(MLLM)上微调适配器,利用近期任务定义奖励模型,引导记忆策略聚焦任务相关内容。为评估方法,我们将VideoMME、EgoLife和EgoTempo重构为流式基准,模拟智能体处理连续观测并在线应对任务的真实场景。评估仅依赖智能体记忆,不访问原始视频。基于Qwen3-VL-30B-A3B,TaskMem在上述基准上分别提升视觉问答准确率6.3%、7.0%和5.3%。

原文摘要 · Abstract (English)

Long-term memory is essential for multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. However, constructing effective memory goes beyond memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize. Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks that simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. To isolate memory assessment, the questions must be answered using only the agent's memory, without access to raw video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.3% on these benchmarks, respectively.

多模态记忆机制强化学习智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。