一个模型同时生成答案并自验证，效率更高更准。
Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization
- 用统一策略联合训练生成与自验证，减少模型数量
- 验证准确率提升34.1%，推理时间降低53.5%
- 适合需要高可靠性推理的视觉语言任务
传统测试时扩展方法需分别训练生成与验证模型，导致训练和推理成本高昂。本文提出优势解耦偏好优化（ADPO），一种统一的强化学习框架，将答案生成与自验证在单一策略中联合学习。ADPO引入两项创新：一是偏好验证奖励，通过正负样本均值验证分数作为决策阈值，当预测正确性与答案正确性一致时给予正反馈；二是解耦优化机制，分别计算生成与验证的优势，使用标记掩码隔离梯度，结合掩码后的GRPO目标，在保持生成质量的同时校准验证分数。ADPO在MathVista/MMMU上准确率分别提升2.8%/1.4%，ReasonSeg上cIoU提升1.9，AndroidControl/GUI Odyssey上步骤成功率提升1.7%/1.0%，验证AUC最高提升34.1%，推理时间降低53.5%。
原文摘要 · Abstract (English)
Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。