让多模态大模型单次前向传播完成精准视觉搜索,又快又可解释。
VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

- 用对数透镜技术从早期隐藏状态读取视觉标记,实现快速定位
- 单次前向传播完成搜索,比基线快8.5到22.2倍
- 兼具速度与可解释性,适合需要实时响应的视觉理解场景
多模态大语言模型在高分辨率图像中定位小物体或稀有目标时存在细粒度视觉搜索难题。现有方法分为两类:(1) 无需训练的基于注意力或置信度的方法准确但慢,需多次查询;(2) 强化学习训练的工具调用模型推理快但不可控、难解释。为此,我们提出 VisLens(通过逻辑透镜实现视觉聚焦),基于对数透镜技术,通过将隐藏状态投影至LLM头部来解码语义。VisLens 还引入轻量级调优透镜,将早期隐藏状态映射到最终隐藏状态空间,使视觉标记可从早期层读出。这些标记与查询中的目标词匹配,生成相关区域裁剪图,并与原图一同输入以生成最终答案。整个流程仅需一次前向传播,无需重复查询。VisLens 在性能上达到或超过已有基线,同时显著降低延迟:比 Thyme 快 8.5–9.9 倍,比训练自由的多轮搜索方法快最多 22.2 倍。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。