解决机器人奖励模型中评分与偏好不一致的问题，提升强化学习性能。
TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
- 通过有序等距校准法修正点评分数，确保与成对偏好一致。
- 在基准测试中，新模型得分达77.96%，接近GPT-5-mini的78.09%。
- 适合需要高一致性奖励信号的长时序机器人任务研究者使用。
奖励模型是具身智能强化学习的瓶颈。长时序机器人操作需要超越手工设计奖励或任务特定标注的可扩展视觉反馈。现有开源视觉语言模型奖励裁判如RoboReward仅采用1-5分轨迹进展评分，缺乏用于RLHF、DPO和Bradley-Terry框架的成对偏好，且未能优化视频场景理解。在RoboReward中引入成对比较与视频问答监督会导致点评分数与成对偏好不一致，引发训练噪声并损害下游性能——此类问题现有聚合方法如TrustJudge无法解决。为此，我们提出TrustRoboReward，一种多范式奖励建模范式，配备偏好有序等距评分编辑（POISE）。我们构建了一个统一的四范式数据集：轨迹进展评分（Score-A）、视频问答质量评分（Score-B）及其对应的成对标签（Pair-A、Pair-B）。成对标签更贴近人类判断，由此启发我们将点点评分校准以避免与成对偏好冲突。POISE修正了点点评分，消除了跨范式评分反转冲突，该问题在TrustJudge上仍存在20.46%。理论上，POISE将评分反转冲突从20.15%降至0%。在基准测试中，基于Qwen3-VL-4B训练的模型获得77.96%的整体奖励得分，几乎媲美GPT-5-mini（78.09%，差距仅0.13%），优于最强的RoboReward-4B基线10.13%。其测试时评分一致性达71.90%，高于RoboReward-4B（57.26%）和GPT-5-mini（68.09%）。推理阶段集成TrustJudge聚合后，总分提升至78.57%，超过GPT-5-mini教师模型。
原文摘要 · Abstract (English)
Reward models are a bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision feedback beyond handcrafted rewards or task-specific annotations. Existing open-source VLM reward judges like RoboReward adopt simple 1--5 trajectory progress scoring, lacking pairwise preferences for RLHF, DPO and Bradley-Terry frameworks, while failing to optimize video scene understanding. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance---an issue aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE). We construct a unified four-paradigm dataset with trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise counterparts (Pair-A, Pair-B). Pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE reduces score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge retains 20.46% conflicts on the same corpus. Evaluated on our benchmark, Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Integrating TrustJudge aggregation during inference boosts the overall score to 78.57%, surpassing the GPT-5-mini teacher model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。