arXiv:2608.01473cs.CVcs.CL2026-08

用少量区域编码替代密集视觉特征,提升手术视觉语言模型效率与可追溯性。

Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

论文配图:Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs
图 1 · 摘自论文原文
  • 用槽位潜变量压缩视觉输入,仅保留关键区域信息。
  • 视觉前缀从1295降至47令牌,总令牌消耗减少91.8%。
  • 支持空间可追溯推理,适合需要定位依据的医疗场景。

用于手术场景理解的多模态大语言模型通常将数百个密集视觉标记注入语言模型,导致计算成本高且生成答案的空间可追溯性差。本文提出Slot2Text,一种双模式手术视觉语言模型,将视觉输入的密集表示替换为一组紧凑的区域编码——槽位潜变量。不同于依赖视觉编码器与语言模型对比对齐,Slot2Text通过自监督方式将视觉特征聚类为少数区域(槽位),这些槽位作为带区域标签的视觉标记被语言模型使用。Slot2Text-Fast利用槽位前缀回答手术问题;Slot2Text-Reason进一步识别并定位推理相关区域,将语言输出与对应槽位、掩码或区域关联。在多个视觉问答与视觉定位基准测试中,Slot2Text-Fast性能媲美现有最优基线,平均总令牌消耗降低91.8%,视觉前缀从1,295减少至47(降幅96.4%)。Slot2Text-Reason以额外令牌和延迟为代价,实现显式的区域身份、位置及可追溯空间证据。结果表明,紧凑槽位潜变量是手术视觉语言模型的高效默认视觉接口,必要时可启用具备空间可追溯性的推理。

原文摘要 · Abstract (English)

Multimodal large language models (MLLM) for surgical scene understanding typically inject hundreds of dense visual tokens into a language model, leading to costly inference and limited spatial traceability for generated answers. We present Slot2Text, a dual-mode surgical MLLM that replaces dense representations of visual input with a compact set of regions encoded as slot latents. Instead of relying on contrastive alignment of the visual encoder with language, Slot2Text groups self-supervised vision features into a few regions--slots that are consumed by the language model as area-labeled visual tokens. Slot2Text-Fast uses the slot prefix to answer surgical questions. Slot2Text-Reason also identifies and locates areas relevant for reasoning, linking language outputs to corresponding slot tokens, masks or regions. Experiments on multiple visual question answering and visual grounding benchmarks show that Slot2Text-Fast is competitive with state-of-the-art baseline at a much lower cost, reducing the average total token consumption by a 91.8\% and the visual prefix from 1,295 to 47 tokens (a 96.4\% reduction). Slot2Text-Reason trades additional tokens and latency for explicit area identities, locations, and traceable spatial evidence. These results establish compact slot latents as an efficient default visual interface for surgical MLLMs, with grounded reasoning invoked when greater spatial traceability is required.

视觉语言模型手术分析高效推理空间可追溯

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。