用排名替代打分,让视频模型更高效地学习图文理解。
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
- 用通用排名器代替专用评分模型,直接比较回复优劣。
- 在多个视频理解任务上超越现有方法,提升显著。
- 适合大规模视频模型对齐,降低人工标注成本。
大型视频-语言模型(VLMs)的进展依赖于大规模微调以增强文本与视觉理解的对齐。主流方法结合监督微调(SFT)与基于偏好数据的强化学习。但随着模型规模扩大,人工反馈成本急剧上升。为降低成本,现有框架采用人工智能反馈的强化学习(RLAIF),以AI作为评判者替代人类偏好。然而,当前方法依赖专门训练的奖励模型,通过视频叙述生成校准的标量奖励,流程复杂且受限。本文提出Oracle-RLAIF,用更通用的Oracle排名器替代训练好的奖励模型,仅需对候选回复进行排序而非打分。同时引入$GRPO_{rank}$,一种基于组相对策略优化的排名损失函数,直接优化序数反馈并利用排名感知优势。实验表明,Oracle-RLAIF在多个视频理解基准上持续优于现有先进VLM方法,为构建灵活、数据高效的多模态视频模型对齐框架提供了新路径。
原文摘要 · Abstract (English)
Recent advances in large video-language models (VLMs) rely on extensive fine-tuning techniques that strengthen alignment between textual and visual comprehension. Leading pipelines typically pair supervised fine-tuning (SFT) with reinforcement learning from preference data to enhance video comprehension. However, as VLMs scale in parameter size, so does the cost of gathering enough human feedback. To make fine-tuning more cost-effective, recent frameworks explore reinforcement learning with AI feedback (RLAIF), which replace human preference with AI as a judge. Current RLAIF frameworks rely on a specialized reward model trained with video narratives to create calibrated scalar rewards -- an expensive and restrictive pipeline. We propose Oracle-RLAIF, a novel framework that replaces the trained reward model with a more general Oracle ranker which acts as a drop-in model ranking candidate model responses rather than scoring them. Alongside Oracle-RLAIF, we introduce $GRPO_{rank}$, a novel rank-based loss function based on Group Relative Policy Optimization (GRPO) that directly optimizes ordinal feedback with rank-aware advantages. Empirically, we demonstrate that Oracle-RLAIF consistently outperforms leading VLMs using existing fine-tuning methods when evaluated across various video comprehension benchmarks. Oracle-RLAIF paves the path to creating flexible and data-efficient frameworks for aligning large multi-modal video models with reinforcement learning from rank rather than score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。