用视觉显著性引导小模型学大模型的多模态推理思路
Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoning
- 用大模型注意力图生成扰动图像，提取关键特征方向
- 通过奇异值分解得到主导引导向量，提升小模型推理能力
- 适合资源受限场景下提升多模态问答的准确性
多模态链式思维（CoT）推理通过逐步推理解析视觉与文本信息。在令牌预算有限的小型模型中，模态间融合常会压制细微的跨模态差异。尤其当不同图像对应相同文本，或相同图像对应不同文本时，融合后输入几乎无法区分。本文提出视觉显著性引导蒸馏（VSSD），利用多模态大语言模型的注意力图生成任务敏感的扰动图像，再通过奇异值分解提取主导引导向量，用于层间知识迁移。在ScienceQA和M³CoT数据集上的实验表明，VSSD显著提升了推理过程生成与答案预测性能。代码已开源。
原文摘要 · Abstract (English)
Multimodal chain-of-thought (CoT) reasoning integrates visual and textual cues through step-by-step inference. In small models with limited token budgets, modality-interaction fusion often suppresses tiny cross-modal differences. In particular, multimodal CoT often struggles when different images pair with identical text or different texts pair with an identical image, making such inputs nearly indistinguishable after fusion. This study proposes Visual Saliency Steering Distillation (VSSD). VSSD leverages the attention maps of multimodal large language models to generate perturbed images that capture task-sensitive feature directions, and then applies singular value decomposition to extract dominant steering vectors to guide inter-layer distillation. Experiments on ScienceQA and M$^3$CoT demonstrate that VSSD improves rationale generation and answer inference. The code is available at https://github.com/BGWH123/VSSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。