用自蒸馏偏好训练，分离多模态学习中的格式与内容，提升模型泛化能力。
Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
- 通过自蒸馏生成偏好数据对，避免依赖大模型或人工标注。
- 偏好训练聚焦格式结构等浅层特征，减少内容记忆，提升泛化性。
- 在多个基准上表现优于基线，尤其在数学和视觉推理任务中提升显著。
基于可验证奖励的强化学习最近推动了“MLLM-r1”方法的发展，将强化学习引入视觉语言模型。现有方法普遍采用监督微调（SFT）作为冷启动，但SFT将推理模式与任务求解、输出格式交织在一起，易导致指令风格过拟合，削弱分布外泛化能力，影响下游强化学习效果。本文从训练方法与数据构建两方面重新审视冷启动问题，提出通用性因子（GF）量化不同方法下的泛化能力。实证发现，基于偏好的训练方法（如DPO）在冷启动阶段泛化性能优于SFT。受此启发，本文提出SPECS——一种自蒸馏的偏好型冷启动框架，实现多模态学习解耦：(1) 通过自蒸馏生成内省式偏好数据对，无需依赖更大教师模型或人工标注；(2) 采用偏好训练学习，专注浅层可迁移的表面形式特征（格式、结构、风格），而非记忆内容；(3) 将学习成果移交至具有可验证奖励的强化学习，以实现深层推理。在多个多模态基准测试中，该框架持续优于强基线，使MEGA-Bench提升4.1%，MathVista提升12.2%。额外实验表明，SPECS有助于降低分布内“卡住”现象，改善探索能力，稳定训练过程，并提升性能上限。
原文摘要 · Abstract (English)
Reinforcement learning (RL) with verifiable rewards has recently catalyzed a wave of "MLLM-r1" approaches that bring RL to vision language models. Most representative paradigms begin with a cold start, typically employing supervised fine-tuning (SFT), to initialize the policy before RL. However, SFT-based cold start adopts the reasoning paradigm intertwined with task solution and output format, which may induce instruction-style overfitting, weakens out-of-distribution generalization, and ultimately affects downstream RL. We revisit the cold start along two views, its training method and data construction, and introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods. Our empirical study finds that preference-based training methods (e.g. DPO) generalizes better than SFT-based methods in cold start. Motivated by this, we propose SPECS-a Self-distilled, Preference-based Cold Start framework that decouples multimodal learning: (1) generates introspective preference data pairs via self-distillation, avoiding reliance on larger teachers or manual annotation; (2) performs preference-based training to learn, focusing on shallow, transferable surface-form criteria (format, structure, style) rather than memorizing content; and (3) hands off to RL with verifiable rewards for deep reasoning results. Experimental results across multiple multimodal benchmarks show that our decoupling learning framework yields consistent performance gains over strong baselines, improving MEGA-Bench by 4.1% and MathVista by 12.2%. Additional experiments indicate that SPECS contributes to reducing in-distribution "stuckness," improving exploration, stabilizing training, and raising the performance ceiling. Project Page: https://kwen-chen.github.io/SPECS-VL/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。