解决多模态模型推理中的模态失衡问题,提升自蒸馏效果
OPD-V: Visual On-Policy Self-Distillation with Modality Balance

- 通过正负模态教师对比,构建平衡性感知的自蒸馏机制
- 在6个基准上平均提升推理准确率,训练成本更低
- 适合优化多模态大模型的后训练阶段,尤其关注模态融合
在多模态大语言模型(MLLM)的后训练中,基于策略的自蒸馏(OPSD)已成为提升视觉推理能力的标准方法。现有方法从多种输入源提取特权信息以指导自蒸馏,但忽略了多模态推理中的固有挑战——模态失衡。当文本信息主导生成时,模型无法充分整合多模态输入,导致精心设计的特权信息被闲置,限制了OPSD的效果。为探究此局限,我们构建了具有不同模态失衡程度的正教师(Zoom-In Image)与负教师(Mask Image)。其推理正确率和词元概率的变化表明,模态平衡本身即可作为特权信息。受此启发,我们提出OPD-V,一种通过正负教师实例化该信息的视觉化OPSD范式。正向模态平衡逻辑差定义了模态平衡信任区域,筛选用于自蒸馏的策略内词元。在6个基准、4个MLLM主干网络及5种后训练方法上的实验表明,OPD-V能持续提升推理性能并降低训练成本。
原文摘要 · Abstract (English)
On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM reasoning. When textual information dominates generation, the model cannot fully integrate its multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their reasoning correctness and token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy tokens used for self-distillation. Experiments across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves reasoning performance while reducing training cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。