开源了一个高效多模态奖励模型，提升视觉语言模型生成质量。
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
- 基于多模态偏好数据训练，对齐人类判断
- 在多模态和纯文本任务上表现优异
- 支持强化学习、测试时选择与数据清洗
尽管大型视觉语言模型（LVLMs）在视觉理解方面表现良好，但偶尔会产生错误输出。虽然带有强化学习或测试时缩放的奖励模型（RMs）有潜力提升生成质量，但一个关键缺口是：公开可用的多模态奖励模型稀缺，且专有模型的实现细节往往不明确。我们通过 InternLM-XComposer2.5-Reward（IXC-2.5-Reward）填补了这一空白，这是一个简单而有效的多模态奖励模型，可将 LVLMs 与人类偏好对齐。为确保 IXC-2.5-Reward 的鲁棒性和通用性，我们构建了一个高质量的多模态偏好语料库，涵盖文本、图像和视频输入，覆盖指令遵循、通用理解、图文文档、数学推理和视频理解等多种领域。该模型在最新的多模态奖励模型基准上取得优异成绩，并在纯文本奖励模型基准上表现具有竞争力。我们进一步展示了三个关键应用：(1) 为强化学习训练提供监督信号；结合近端策略优化（PPO）得到 IXC-2.5-Chat，显著提升指令遵循和多模态开放对话能力；(2) 在测试时从候选响应中选择最优结果；(3) 过滤现有图像和视频指令微调数据中的异常或噪声样本。为确保可复现性并促进后续研究，我们已在 https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward 开源所有模型权重和训练方案。
原文摘要 · Abstract (English)
Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. To ensure reproducibility and facilitate further research, we have open-sourced all model weights and training recipes at https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-Reward
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。