通过遮蔽推理前缀,让小模型更依赖视觉信息提升多模态推理能力。
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation

- 用关键推理前缀遮蔽策略引导学生模型关注视觉证据。
- 在多个基准上超越开源模型和自蒸馏方法,推理准确率更高。
- 适合需要高效视觉推理的轻量级多模态模型研发者。
近期的多模态大模型(VLM)思考-作答方法(如 Qwen3-VL-Thinking)通过中间推理步骤提升推理性能,但计算开销显著,尤其对大型模型而言。为将此类能力蒸馏到紧凑的思考-作答 VLM,核心挑战在于增强学生模型在整个推理过程中的视觉证据利用能力,因长推理轨迹易出现视觉遗忘。为此,本文提出一种新型思考-作答蒸馏框架,通过遮蔽学生模型的关键推理前缀,强制其将推理锚定在视觉信息上。为弥补被遮蔽的文本线索,学生在蒸馏过程中被鼓励更多依赖视觉证据。提出的遮蔽策略包括:1)基于注意力贡献的逐标记关键推理前缀遮蔽,针对每个下一个词预测选择性屏蔽高影响力前缀;2)自适应遮蔽预算调度,根据教师-学生分布差异动态增加遮蔽比例。蒸馏阶段采用本方法设计的显著推理前缀掩码,替代标准因果掩码,同时阻断未来词与关键推理线索。实验表明,该方法在多模态推理基准上优于现有开源 VLM、VLM 蒸馏与自蒸馏方法,且分析证实学生模型在推理全程中视觉利用率显著提升。
原文摘要 · Abstract (English)
Recent think-answer approaches in VLMs, such as Qwen3-VL-Thinking, boost reasoning performance by leveraging intermediate thinking steps before the final answer, but their computational cost becomes substantial, especially for larger VLMs. To distill such capabilities into compact think-answer VLMs, a primary objective is to improve the student's ability to utilize visual evidence throughout its reasoning trace, as long think-answer traces suffer from visual forgetting issues. To this end, we introduce a novel think-answer distillation framework that encourages the student to anchor its thinking on visual information by masking the student's salient reasoning prefixes. To compensate for such masked textual cues, the student is encouraged to rely more on visual evidence as an alternative source of information during distillation. Our masking strategies include: 1) token-wise salient reasoning-prefix masking, which masks high-influence reasoning prefixes selectively for each next-token prediction, and 2) self-paced masking budget scheduling, which gradually increases the masking scale according to distillation difficulty, measured by the discrepancy between teacher--student distributions. In the distillation phase, the student is guided by our salient reasoning-prefix mask, which blocks both future tokens and salient reasoning cues, in place of the standard causal mask used for auto-regressive language modeling. Experimental results show that our approach outperforms recent open-source VLMs, VLM distillation, and self-distillation methods on multimodal reasoning benchmarks, while further analyzes confirm enhanced visual utilization along the student thinking process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。