ACPO提升视觉语言模型复杂推理对齐效果,解决训练僵化与更新不稳问题。
ACPO: Adaptive Curriculum Policy Optimization for Aligning Vision-Language Models in Complex Reasoning
- 动态课程机制分阶段优化探索与利用,逐步增加样本复用。
- 自适应裁剪机制按每个标记的优势值调整更新幅度,提升学习效率。
- 在数学与逻辑推理任务上表现最优,适合高阶多模态模型训练者。
通过强化学习对齐大规模视觉语言模型进行复杂推理,常受现有策略优化算法限制,如固定训练周期和PPO中僵化的统一裁剪机制。本文提出自适应课程策略优化(ACPO),采用双组件自适应学习策略。首先,ACPO设计动态课程,通过逐步增加样本复用,实现从稳定近策略探索到高效离策略利用的有序过渡;其次,提出优势感知自适应裁剪(AAAC)机制,以各标记的归一化优势为依据动态设定裁剪边界,取代固定超参数,使高潜力样本获得更大梯度更新,同时抑制有害更新。我们在MathVista、LogicVista和MMMU-Pro等多个高难度多模态推理基准上进行大量实验,结果表明ACPO持续优于DAPO、PAPO等强基线,达到当前最佳性能,收敛更快且训练更稳定。
原文摘要 · Abstract (English)
Aligning large-scale vision-language models (VLMs) for complex reasoning via reinforcement learning is often hampered by the limitations of existing policy optimization algorithms, such as static training schedules and the rigid, uniform clipping mechanism in Proximal Policy Optimization (PPO). In this work, we introduce Adaptive Curriculum Policy Optimization (ACPO), a novel framework that addresses these challenges through a dual-component adaptive learning strategy. First, ACPO employs a dynamic curriculum that orchestrates a principled transition from a stable, near on-policy exploration phase to an efficient, off-policy exploitation phase by progressively increasing sample reuse. Second, we propose an Advantage-Aware Adaptive Clipping (AAAC) mechanism that replaces the fixed clipping hyperparameter with dynamic, sample-wise bounds modulated by the normalized advantage of each token. This allows for more granular and robust policy updates, enabling larger gradients for high-potential samples while safeguarding against destructive ones. We conduct extensive experiments on a suite of challenging multimodal reasoning benchmarks, including MathVista, LogicVista, and MMMU-Pro. Results demonstrate that ACPO consistently outperforms strong baselines such as DAPO and PAPO, achieving state-of-the-art performance, accelerated convergence, and superior training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。