用连续奖励机制提升多模态大模型自进化推理稳定性。
Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling
- 引入重推断机制拓展长尾推理路径探索
- 用连续信号替代二值奖励，按频次校准反馈
- 适合关注多模态推理与自进化方法的研究者
在多模态大语言模型的无监督自进化中，后训练阶段的反馈信号质量对学习稳定性与有效性至关重要。现有方法主要依赖多数投票选取最频繁输出作为伪黄金答案，但可能源于模型固有偏见而非客观正确性。为此，我们提出连续软化重采样（CSRS）方法。具体包括：重推断重构机制（RRM），使模型从锚点重新推理，扩展长尾推理路径的探索；软化频率奖励（SFR），以连续信号替代二值奖励，根据采样推理集中的答案频率动态校准奖励；结合视觉语义扰动（VSP），确保模型优先关注数学逻辑而非视觉表层特征。实验表明，CSRS显著提升了Qwen2.5-VL-7B在MathVision等基准上的推理性能，在几何任务上达到无监督自进化新最佳水平。代码已开源。
原文摘要 · Abstract (English)
In the unsupervised self-evolution of Multimodal Large Language Models, the quality of feedback signals during post-training is pivotal for stable and effective learning. However, existing self-evolution methods predominantly rely on majority voting to select the most frequent output as the pseudo-golden answer, which may stem from the model's intrinsic biases rather than guaranteeing the objective correctness of the reasoning paths. To counteract the degradation, we propose Continuous Softened Retracing reSampling (CSRS) in MLLM self-evolution. Specifically, we introduce a Retracing Re-inference Mechanism (RRM) that the model re-inferences from anchor points to expand the exploration of long-tail reasoning paths. Simultaneously, we propose Softened Frequency Reward (SFR), which replaces binary rewards with continuous signals, calibrating reward based on the answers' frequency across sampled reasoning sets. Furthermore, incorporated with Visual Semantic Perturbation (VSP), CSRS ensures the model prioritizes mathematical logic over visual superficiality. Experimental results demonstrate that CSRS significantly enhances the reasoning performance of Qwen2.5-VL-7B on benchmarks such as MathVision. We achieve state-of-the-art (SOTA) results in unsupervised self-evolution on geometric tasks. Our code is avaible at https://github.com/yyy195/CSRS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。