arXiv:2606.07000cs.AI2026-06

用隐式提示指导模型推理,提升多模态任务表现。

Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

论文配图:Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
图 1 · 摘自论文原文
  • 通过空间注意力和中间推理步骤生成隐式提示,提供逐步监督。
  • 在2B到8B参数模型上显著优于基线,减少熵坍缩现象。
  • 适合需要高效推理且避免答案泄露的多模态强化学习场景。

近期基于后训练的方法,尤其是可验证奖励强化学习(RLVR),显著提升了大视觉语言模型(LVLM)的推理能力。然而,可验证奖励的稀疏性难以对失败的推理轨迹提供细粒度监督,导致复杂多模态推理任务中探索效率低下。虽然策略蒸馏能提供密集指导,但外部教师方法带来巨大计算开销,而基于答案的微调可能暴露答案信息并引发捷径生成行为。为此,我们提出PTD-PO框架,一种无需暴露答案的特权教学蒸馏策略优化方法。该方法从空间注意力引导与中间文本推理步骤构建结构化特权提示,通过上下文学习生成逐令牌分布的监督信号。学生模型仍在原始无答案环境中优化,其失败轨迹在令牌分布层面与提示增强的参考模型对齐。为应对有引导与无引导情境间的分布偏移,引入Top-K Jensen-Shannon散度目标,聚焦于高信息量的令牌概率,同时降低内存开销。在2B至8B参数的LVLM上实验表明,PTD-PO持续优于RLVR与蒸馏基线,缓解熵坍缩,提升复杂多模态推理性能。

原文摘要 · Abstract (English)

Recent post-training methods, particularly Reinforcement Learning with Verifiable Rewards (RLVR), have significantly enhanced the reasoning ability of Large Vision-Language Models (LVLMs). However, the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks. Although policy distillation can offer dense guidance, external teacher based methods introduce substantial computational overhead, while answer conditioned tuning methods may expose answer-level information and induce shortcut-like generation behavior. To address these limitations, we propose PTD-PO, a Privileged Tutoring Distillation Policy Optimization framework for RLVR that provides dense guidance without exposing the answer to the student policy. Specifically, PTD-PO constructs structured privileged hints from spatial attention guidance and intermediate textual reasoning steps, and uses them through in-context learning to produce step-wise token-distribution supervision. The student is still optimized under the original answer-free context, and its failed rollouts are aligned with the hint-augmented reference model at the token-distribution level. To further stabilize distillation under the distribution shift between guided and unguided contexts, we introduce a Top-K Jensen-Shannon divergence objective that focuses alignment on informative token probabilities while reducing memory overhead. Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines, mitigates entropy collapse, and improves complex multimodal reasoning performance.

多模态强化学习蒸馏推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。