提出无需训练的视觉记忆管理器,让长时序智能体直接保留原始视觉信息。
When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents
- 用确定性渲染将历史转化为视觉表征,不依赖显式记忆操作
- 在多模态任务上比其他方法准确率更高,文本任务上保持竞争力
- 无需训练即可减少31.5%~63.1%的非缓存令牌,适合长期决策场景
长时序智能体需通过上下文管理器将不断增长的交互历史压缩为有限工作空间。现有光学记忆研究多将像素视为文本化历史的密集编码器,常假设视觉渲染会显著降低性能,因而需结合SFT、自蒸馏或强化学习来弥补差距。本文提出在统一框架下公平比较视觉渲染(VR)作为上下文管理器的表现,并探讨其在本源多模态历史中的天然优势。我们构建了无训练的VERA策略:在文本任务中将文本历史转为视觉表征,在多模态任务中直接保留原始视觉观测,避免转译。在四个文本和三个多模态基准上,VERA相较无压缩基线减少31.5%-63.1%非缓存令牌,文本任务表现相当,多模态任务达到最高准确率,验证了保持模态原生性的上下文管理有效性。
原文摘要 · Abstract (English)
Long-horizon agents need a context manager to compress growing interaction histories into a bounded working context, via passive strategies or active strategies that decide how memory is accessed and reorganized. Meanwhile, prior optical-memory work mainly treats pixels as a dense codec for textualized histories, often presupposing that rendering context into optical memory incurs a significant performance drop relative to text, thus coupling this representation with SFT, self-distillation, or reinforcement learning to close this gap, leaving unresolved (i) how visual rendering performs as a context manager under a fair, controlled comparison, and (ii) whether this carrier offers a native advantage when history is inherently multimodal. In this paper, we formulate context management as a budget-constrained history transformation and introduce Visual Rendering (VR) as a representational context manager. Under a shared harness, policy model, trigger, and task domain, we evaluate VR on four text-centric and three multimodal benchmarks against four baselines (No Compression, Discard-All, Sliding Window, Summarization), finding visual memory is a natural carrier of native visual evidence. Building on this finding, we propose VERA (Visual Evidence-Retaining strategy for long-horizon Agents), a training-free context manager built on deterministic rendering with no exposed memory operations: on text-centric benchmarks it renders textual history as VR does, while on multimodal benchmarks it retains native visual observations instead of translating them into text. Across nearly all benchmarks, VERA cuts cumulative non-cache tokens by 31.5%-63.1% versus No Compression, matches existing managers on text-centric tasks, and achieves the highest accuracy among all baselines on multimodal tasks, supporting a modality-preserving view of long-horizon context management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。