用全局归一化解决多模态推理中强化学习的梯度不稳定问题
Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization
- 将原始KL分数转为批次级相对优势，稳定训练过程
- 在多个多模态推理任务上显著提升模型性能与训练鲁棒性
- 适合需要稳定微调的多模态大模型研究者
近期，基于策略的蒸馏（OPD）成为重要的后训练范式。通过强教师模型为采样轨迹提供密集、细粒度的监督，OPD相比依赖稀疏二元或结果反馈的强化学习（RLVR）具有明显优势。然而，简单的逐标记蒸馏易受异常状态中幅度不匹配导致的梯度不稳定影响。为此，我们提出全局归一化蒸馏策略优化（GNDPO），通过将原始KL得分转换为批次级相对优势来稳定优化。该归一化有效缓解梯度爆炸，同时保留逐标记指导的优势。实验表明，GNDPO在多个多模态推理任务上显著提升训练鲁棒性与下游性能。代码已开源：https://github.com/OPPO-Mente-Lab/GNDPO。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has recently emerged as an important post-training paradigm. By using a stronger teacher model to provide dense, fine-grained supervision for sampled trajectories, OPD offers a clear advantage over reinforcement learning with verifiable rewards (RLVR), which typically depends on sparse binary or outcome-based environmental feedback. However, naive token-level distillation can suffer from gradient instability, due to magnitude misalignment in outlier states. To address this issue, we propose Globally Normalized Distillation Policy Optimization (GNDPO), a practical method that stabilizes optimization by transforming raw KL scores into batch-level relative advantages. This normalization effectively mitigates gradient explosions while retaining the benefits of token-level guidance. Experimental results show that GNDPO substantially improves training robustness and downstream performance across multimodal reasoning tasks. The code is released at https://github.com/OPPO-Mente-Lab/GNDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。