用大模型自动生成高质量奖励数据，提升视觉语言模型对齐效果
From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
- 用图像描述作为弱监督信号筛选高置信度数据，指导模型微调
- 利用低置信度数据生成多样偏好样本，构建可靠优选-拒选对
- 在多个评测基准上优于传统方法，适合追求高精度对齐的研究者
将大型视觉语言模型（LVLM）与人类偏好对齐面临细粒度、高质量且多模态偏好数据稀缺的问题，现有基于直接蒸馏的方法常因低置信度数据导致性能不佳。为此，我们提出CAREVL，一种通过可靠利用高/低置信度数据增强奖励建模的新方法。首先，一组辅助专家模型（文本奖励模型）创新性地利用图像描述作为弱监督信号，筛选出高置信度数据，并用于微调LVLM；其次，将低置信度数据用于生成多样化偏好样本，经微调后的模型评分并筛选，构建可靠的优选-拒选对以进一步训练。CAREVL在VL-RewardBench和MLLM-as-a-Judge基准上均优于传统蒸馏方法，证明其有效性。代码即将开源。
原文摘要 · Abstract (English)
Aligning large vision-language models (LVLMs) with human preferences is challenging due to the scarcity of fine-grained, high-quality, and multimodal preference data without human annotations. Existing methods relying on direct distillation often struggle with low-confidence data, leading to suboptimal performance. To address this, we propose CAREVL, a novel method for preference reward modeling by reliably using both high- and low-confidence data. First, a cluster of auxiliary expert models (textual reward models) innovatively leverages image captions as weak supervision signals to filter high-confidence data. The high-confidence data are then used to fine-tune the LVLM. Second, low-confidence data are used to generate diverse preference samples using the fine-tuned LVLM. These samples are then scored and selected to construct reliable chosen-rejected pairs for further training. CAREVL achieves performance improvements over traditional distillation-based methods on VL-RewardBench and MLLM-as-a-Judge benchmark, demonstrating its effectiveness. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。