让视觉编码器记住之前图像，提升多图对比能力。
Stateful Visual Encoders for Vision-Language Models
- 视觉编码器引入状态记忆，基于前序图像特征生成当前表示
- 在跨图像空间聚合等任务上显著提升，尤其对细微变化敏感
- 适用于医学影像、遥感等需长期视觉理解的场景
视觉语言模型（VLMs）在多图像、多轮交互任务中日益重要，其决策依赖于视觉变化。然而现有开放权重VLMs中，视觉比较仅发生在语言模型内，视觉编码器本身无状态：每张图像独立编码，无法获取先前视觉上下文。导致微小但关键的变化在语言模型分析前已被削弱，尤其当这些变化不改变场景高层语义时。本文提出状态化视觉编码器，使每个视觉表征依赖于之前的视觉特征。在监督微调下，配备该编码器的VLM在涉及跨图像空间聚合、多对象视觉差异检测和视觉轨迹行为克隆的任务中持续取得提升，且在不同输入分辨率、语言模型规模和VLM主干架构下均有效。最终在真实世界任务中验证：包括纵向放射学、细粒度图像比较和遥感，状态化编码器持续优于通用基线，并在部分领域达到或超过专用模型性能。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the language model, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the language model has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions, language model sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。