构建可长期记忆的多模态智能体，解决视觉信息爆炸与跨轮引用难题。
Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
- 用情景化视觉记忆外存替代全量输入，按需召回关键信息
- 20轮对话下检索准确率达91.4%，推理速度提升近一半
- 适合需要长时记忆与复杂交互的多模态应用开发者
统一多模态模型虽能联合完成视觉理解、语言理解与图像生成编辑，但受限于将所有历史视觉和文本输入反复送入共享上下文窗口，导致视觉令牌激增，跨轮引用不可靠。本文提出认知结构化多模态智能体（Cognitive-structured Multimodal Agent），将视觉信息外置于情景化视觉记忆中，并在推理时选择性激活相关片段。该智能体包含感知抽象引擎、认知检索引擎与多模态执行控制器。为解决现有数据集缺乏逐轮检索监督的问题，构建了统一场景引擎，自动生成带细粒度检索标注的多轮对话，支持强化学习优化抽象与检索策略。同时构建分难度层级的长时程视觉对话基准以评估情景记忆召回能力。8B规模智能体在20轮会话中达到91.4%检索准确率，优于32B基线8.2个百分点，且每轮推理时间从23.1秒降至12.7秒。进一步推出认知结构化多模态智能体工具包（CMA-Harness），集成持久化多模态记忆、网络访问、图像生成/编辑/合成工具及OpenAI兼容服务。结构化记忆与模块化决策提供比单纯参数扩展更高效、可扩展的长时程多模态智能体范式。
原文摘要 · Abstract (English)
Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。