给视觉语言模型加记忆模块，让其生成更连贯准确。
VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
- 设计短期感知记忆与长期语义记忆双模块，模拟人类认知过程。
- 在多个视觉任务上平均提升11.0%，显著优于基线模型。
- 适合需要长程理解与生成一致性的复杂视觉任务研究者。
尽管视觉语言模型（VLMs）取得了显著成功，但在一系列复杂视觉任务上的表现常受'视觉处理瓶颈'制约：在长时间生成过程中容易丢失视觉证据的锚定，缺乏情境化的视觉体验。受人类认知记忆理论启发——区分以视觉为主导的短期记忆和以语义为主导的长期记忆——我们提出VisMem，一种认知对齐的框架，为VLMs引入动态潜在视觉记忆：一个用于精细感知保留的短期模块，以及一个用于抽象语义整合的长期模块。这些记忆在推理过程中无缝调用，使VLMs能够在思考与生成过程中同时保持感知保真度与语义一致性。在涵盖理解、推理与生成的多样化视觉基准上的大量实验表明，VisMem相比原模型平均性能提升11.0%，全面超越所有对比方法，确立了潜在空间记忆增强的新范式。
原文摘要 · Abstract (English)
Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。