arXiv:2608.23268cs.CV2026-08

用双粒度记忆增强冻结模型,提升多模态推理能力

Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner

  • 构建实例与规则两类记忆,通过反射抽象生成规则
  • 在多个测试集上显著优于无记忆和现有基线方法
  • 适合无法微调的闭源或设备端模型使用

前沿多模态大语言模型(MLLM)在感知任务表现优异,但在科学与数学推理方面仍显不足。参数级微调对闭源或设备端模型不可行,而无状态提示无法积累已解决问题的经验。本文提出DG-Mem:一种双粒度代理记忆框架,通过训练阶段一次性构建、测试时只读访问的非参数外部记忆,增强冻结的MLLM。受人类记忆互补学习系统启发,记忆分为基于实例的样例记忆和基于类别规则的IF-THEN记忆,中间通过瞬时反思存储实现分离构建,确保规则仅由抽象反思生成,不直接依赖样例文本。两个关键设计:在线概念分类器在训练中增量扩展类别空间,无需预设分类体系;基于谢尔普利值的上下文归因机制,分解规则集贡献度,生成每条规则的效用权重,用于测试时重加权检索。整个流程不引入梯度更新,可部署于闭源或设备端模型。在MathVista、MMMU、MMMU-Pro三个数据集上,针对四种开放权重与专有模型(Qwen3.5-27B、Qwen3.5-122B-A10B、GPT-5-Nano、Gemini-3-Flash),DG-Mem均一致优于无记忆基线与竞争性记忆方法。

原文摘要 · Abstract (English)

Frontier multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning. Parameter-level adaptation is unavailable for closed-weight or on-device backbones, and stateless prompting forfeits any compounding benefit from problems already solved. We propose \textbf{DG-Mem}, a dual-grained agentic memory framework that augments a frozen MLLM with a non-parametric, externally stored memory built once from training-time rollouts and consulted read-only at test time. Motivated by the Complementary Learning Systems (CLS) account of human memory, DG-Mem factors its store into an instance-grounded exemplar memory and a category-level schema memory of IF-THEN rules, with a transient reflection store mediating their construction so that schemas are synthesized only from abstract reflections, never from exemplar text. Two design choices distinguish DG-Mem: an online concept categorizer that grows the category space incrementally during training rather than committing to a predefined taxonomy, and a Shapley context attribution procedure that decomposes correctness across the entire retrieved rule set and yields a per-rule utility that re-weights retrieval at test time. The pipeline introduces no gradient updates and is deployable on closed-weight or on-device backbones. Across MathVista, MMMU, and MMMU-Pro on four open-weight and proprietary backbones (Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, Gemini-3-Flash), DG-Mem improves consistently over no-memory and competitive memory baselines.

多模态推理记忆机制冻结模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。