用自动学习的评分规则生成可解释的图像生成评估模型,仅需极少量标注数据。
AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment

- 从偏好对中自动生成评分规则,由视觉语言模型评估并筛选最优规则
- 仅用不到0.01%的人工标注数据,便达到主流奖励模型性能
- 规则可解释性强,适合需要透明评估的文本到图像生成任务
文本到图像(T2I)生成模型与人类偏好对齐越来越依赖于基于图像的奖励模型,该模型根据提示一致性和感知质量对生成图像进行评分或排序。现有奖励模型通常在大规模人工偏好语料库上训练为布拉德利-特里(BT)偏好模型,导致训练成本高、难以适配且评估标准不透明。相比之下,视觉语言模型(VLM)可通过文本评分规则提供更细粒度的评估,但其手动设计或启发式生成的评分规则可能无法可靠反映人类偏好。本文提出AutoRubric-T2I,首个在T2I领域实现评分规则自动学习的框架,能够自动合成并选择显式评分规则以指导VLM判官。该方法首先将偏好对中的推理轨迹转化为候选规则,再通过VLM在每条规则下评分成对图像,生成规则得分差用于偏好学习。为去除噪声和冗余规则,进一步采用ℓ₁正则化逻辑回归精炼器,选出前N个最具区分性的规则。大量实验表明,AutoRubric-T2I仅需不到0.01%的标注偏好数据即可生成高质量、可解释的奖励信号,显著降低对大规模奖励模型训练的需求。在MMRB2等图像奖励基准上,其性能优于多个强基线奖励模型。进一步验证其作为强化学习奖励在下游T2I任务(如TIIF和UniGenBench++)中的有效性,在扩散模型的Flow-GRPO流程中,其生成质量优于标量奖励模型。
原文摘要 · Abstract (English)
Aligning Text-to-Image (T2I) generation models with human preferences increasingly relies on image reward models that score or rank generated images according to prompt alignment and perceptual quality. Existing reward models are commonly trained as Bradley-Terry (BT) preference models on large-scale human preference corpora, making them costly to train, difficult to adapt, and opaque in their evaluation criteria. Meanwhile, Vision-Language Model (VLM) judges can provide more fine-grained assessments through textual rubrics, but their manually designed or heuristically generated scoring rules may fail to reliably reflect human preferences. In this paper, we propose AutoRubric-T2I, the first rubric learning framework in T2I that automatically synthesizes and selects explicit rubrics for guiding VLM judges. AutoRubric-T2I first synthesizes reasoning traces from preference pairs into candidate rubrics, then uses a VLM judge to score paired images under each rubric, producing pairwise rubric-score differences for preference learning. To remove noisy and redundant rules, we further employ a $\ell_1$-Regularized Logistic Regression Refiner, which selects the Top-$N$ most discriminative rubrics. Extensive evaluations show that AutoRubric-T2I produces high-quality, interpretable reward signals using less than 0.01% of the annotated preference data, substantially reducing the need for large-scale reward-model training. On image reward benchmarks such as MMRB2, AutoRubric-T2I outperforms strong reward model baselines. We further validate AutoRubric-T2I as an RL reward on downstream T2I tasks, including TIIF and UniGenBench++, where it improves generation quality over scalar reward models using the Flow-GRPO pipeline on diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。