让视觉语言模型高效记住历史，提升长期决策能力。
Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models
- 用动态压缩的固定内存保存历史信息，不拼接图像帧
- 注意力计算量减少25%以上，内存节省13%，长时任务表现更好
- 适合需要连续理解的场景，如视频分析、机器人决策
在序列决策任务中，历史上下文融合是视觉语言模型（VLM）面临的核心挑战。现有VLM独立处理视觉输入，导致难以实现时间理解。直接将历史帧接入Transformer会引发二次方级注意力复杂度和过高内存开销。现有方法或计算成本激增，或通过时间压缩造成显著信息丢失。为此，我们提出动态上下文适配器（DCA），一种针对预训练VLM的新颖上下文注入方法。该方法采用固定大小、动态压缩的记忆模块，在不进行帧拼接的前提下保留历史语义。DCA连接静态VLM与循环策略，使预训练模型具备记忆能力，同时保持计算高效。实验表明，DCA在长时任务上实现超过25%的注意力FLOPs降低和13%的内存节省，同时性能提升。
原文摘要 · Abstract (English)
Historical context integration presents a fundamental challenge for Vision-Language Models (VLMs) in sequential decision-making tasks. Current VLMs process visual inputs independently, which creates critical limitations for downstream applications that require temporal understanding. Direct incorporation of historical frames into Transformer inputs produces quadratic attention complexity and excessive memory consumption. Existing approaches suffer from significant drawbacks: computational inflation or substantial information loss through temporal compression. To address these challenges, we introduce Dynamic Context Adapter (DCA), a novel context injection approach for pretrained VLMs. Our method employs fixed-size, dynamically compressed memory to preserve historical semantics without frame concatenation. DCA bridges static VLMs and recurrent policies and enables memory capabilities in pretrained models while maintaining computational efficiency. DCA achieves over $25\%$ reduction in attention FLOPs and $13\%$ memory savings while improving performance on long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。