用高差异提示提升多模态模型蒸馏效果,突破低效瓶颈
DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation

- 基于答案分布差异设计新提示,避免无效训练
- 合成20万条高差异提示,使模型性能提升最高15%
- 适合需要高效蒸馏的多模态推理模型研究者
蒸馏使紧凑的视觉-语言模型获得强大推理能力,但现有提示多依赖简单启发式或现成数据集。我们发现:标准图表/文档推理数据集中高达69%的提示为零差异(zero-delta),即教师与学生输出分布完全一致,训练无法提供有效学习信号,导致学生性能快速饱和。为此,我们回归本质——蒸馏旨在最小化分布差异,只有揭示师生能力差距的提示才具价值。通过量化答案差异(Δ),我们证明非零差异对有效扩展至关重要。据此提出分阶段合成流程,以现有数据为种子,主动针对学生薄弱环节生成高质量提示。最终构建出包含20万条合成高差异推理问题的DeltaPrompts数据集。在三种场景下评估均表现优异:在目标师生对上进行在线蒸馏、迁移到新模型家族无需重生成数据、以及非推理模型的离线微调。整体平均在10个涵盖图表、文档和感知推理的基准上实现最高15%相对提升,即使在高度优化的Qwen3-VL-8B-Thinking模型上仍显著增益。
原文摘要 · Abstract (English)
Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities, yet the prompts driving this process are typically chosen via simple heuristics or aggregated from off-the-shelf datasets. We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta, meaning the teacher and student already induce the exact same answer distribution. Training on these prompts provides minimal learning signal, causing student improvement to rapidly saturate regardless of data scale. To escape the zero-delta trap, we return to first principles: distillation fundamentally minimizes distributional divergence, and thus a prompt is valuable only if it exposes a functional capability gap between the teacher and student. We quantify this gap through answer divergence ($Δ$), demonstrating that non-zero divergence is critical for effective scaling. Building on this insight, we propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts. The result is DeltaPrompts, a diverse dataset of 200k synthetic, high-divergence reasoning problems. We evaluate DeltaPrompts across three distinct settings: on-policy distillation with the target teacher-student pair, transfer to a novel model family without regenerating the data, and off-policy fine-tuning of a non-reasoning model. Across all scenarios, DeltaPrompts drives substantial gains, yielding up to 15% relative improvement even on top of a highly-optimized reasoning model (e.g., Qwen3-VL-8B-Thinking) -- averaged over 10 benchmarks spanning chart, document and perception-centric reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。