arXiv:2511.06722cs.CVcs.AI2025-11AAAI

通过难易度区分采样,提升多模态模型后训练效果

Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View

  • 根据图像退化和跨模态注意力分析,量化样本难易度
  • 基于难易分层采样,GRPO训练比传统方法准确率更高
  • 适合研究多模态推理与强化学习融合的学者

多模态大语言模型在链式思维推理方面取得进展。受Deepseek-R1启发,研究者将强化学习后训练扩展至多模态推理,主要集中于数学数据集。但现有后训练范式忽视两点:(1) 缺乏可量化的难易度指标,难以筛选优化样本;(2) 后训练策略未能协同优化感知与推理能力。为此,我们提出两种难度感知采样策略:渐进式图像语义掩码(PISM)通过系统性图像退化量化样本难度,跨模态注意力平衡(CMAB)通过注意力分布分析评估跨模态交互复杂度。基于这些指标,设计分层训练框架,包含仅GRPO与SFT+GRPO混合训练,并在六个基准数据集上评估。实验表明,对难度分层样本应用GRPO优于传统SFT+GRPO流程,证明战略性数据采样可替代监督微调并提升模型准确率。代码将发布于https://github.com/qijianyu277/DifficultySampling。

原文摘要 · Abstract (English)

Recent advances in Multimodal Large Language Models (MLLMs) have spurred significant progress in Chain-of-Thought (CoT) reasoning. Building on the success of Deepseek-R1, researchers extended multimodal reasoning to post-training paradigms based on reinforcement learning (RL), focusing predominantly on mathematical datasets. However, existing post-training paradigms tend to neglect two critical aspects: (1) The lack of quantifiable difficulty metrics capable of strategically screening samples for post-training optimization. (2) Suboptimal post-training paradigms that fail to jointly optimize perception and reasoning capabilities. To address this gap, we propose two novel difficulty-aware sampling strategies: Progressive Image Semantic Masking (PISM) quantifies sample hardness through systematic image degradation, while Cross-Modality Attention Balance (CMAB) assesses cross-modal interaction complexity via attention distribution analysis. Leveraging these metrics, we design a hierarchical training framework that incorporates both GRPO-only and SFT+GRPO hybrid training paradigms, and evaluate them across six benchmark datasets. Experiments demonstrate consistent superiority of GRPO applied to difficulty-stratified samples compared to conventional SFT+GRPO pipelines, indicating that strategic data sampling can obviate the need for supervised fine-tuning while improving model accuracy. Our code will be released at https://github.com/qijianyu277/DifficultySampling.

多模态后训练强化学习采样策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。