检验视觉语言模型是否真用连续思维令牌进行推理。
Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
- 提出可标准化的令牌替换测试,隔离内容与存在性影响。
- 多数模型在令牌内容被替换后仍保持高准确率。
- 揭示了有通道不等于真使用,适合评估新推理机制的论文。
视觉语言模型(VLMs)越来越多地引入连续或隐式非文本令牌以支持“视觉思考”。尽管任务准确率提升,但这并不足以证明模型真正利用这些令牌进行推理——性能增益可能源于上下文长度增加、特殊令牌锚定或训练时正则化等混淆因素。本文提出诊断原则“Ablate-to-Validate”,并实现为令牌替换测试(TRT),通过固定提示、图像、令牌预算和解码方式,将中间令牌替换为零值、随机值、首次重复值或理想值,从而检验性能是否依赖于令牌内容而非仅存在。作为对照实验,研究在LLaVA-13B和Qwen2.5-VL-3B上预测与消耗连续或离散深度跨度的表现,采用多个冻结编码器(SigLIP2、CLIP、DINOv2)和不同令牌预算。此外,还将TRT应用于三个现成的视觉思考系统(Mirage、Mull-Tokens、CoVT)在BLINK、VSP和CV-Bench数据集上的表现。结果表明,准确率提升是虚假代理:即使令牌内容被破坏或替换,模型仍保留大部分性能增益,暴露出拥有隐式通道与真正将其作为信息瓶颈之间的巨大差距。建议在未来引入连续思维令牌的方法中,将TRT作为与准确率并列的标准诊断工具。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly augmented with continuous or latent non-textual tokens intended to support "visual thinking." Despite improved task accuracy, this alone does not show that models actually use these tokens for reasoning -- gains may arise from confounds such as added context length, special-token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, for testing whether latent-token content is genuinely utilized, and instantiate it as the Token Replacement Test (TRT), a standardized suite of content-replacement ablations. TRT holds the prompt, image, token budget, and decoding fixed while replacing intermediate tokens with zero, random, first-repeat, or oracle alternatives, isolating whether performance depends on token content or merely on token presence. As a controlled testbed, we study relative depth reasoning with LLaVA-13B and Qwen2.5-VL-3B, training models to predict and consume continuous or discrete depth spans across multiple frozen encoders (SigLIP2, CLIP, DINOv2) and token budgets. We additionally apply TRT to three off-the-shelf visual-thinking systems (Mirage, Mull-Tokens, CoVT) on BLINK, VSP, and CV-Bench. Across all settings, accuracy gains are a misleading proxy for latent-token reasoning: VLMs retain most improvement even when token content is corrupted or replaced, revealing a persistent gap between having a latent channel and using it as an information bottleneck. We recommend TRT as a standard diagnostic alongside accuracy for any method introducing continuous thought tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。