用100个示例自进化出图像编辑评估能力,比大模型更省数据。
RewardHarness: Self-Evolving Agentic Post-Training

- 通过工具库迭代演化替代传统参数训练
- 仅用0.05%数据达到47.4%评估准确率
- 适合低资源场景下的智能评估系统构建
评估指令引导的图像编辑需要反映细微人类偏好的奖励机制,但现有奖励模型通常依赖大规模偏好标注和额外训练,造成数据效率差距:人类仅需少量示例即可推断目标评价标准,而模型常需数十万次对比训练。我们提出RewardHarness,一种自进化代理式奖励框架,将奖励建模重构为上下文演化而非权重优化。该框架不依赖大规模标注,而是通过仅100个偏好示范,迭代演化一套工具与技能库以对齐人类偏好。给定源图、候选编辑图及编辑指令,调度器从维护的工具库中选取最相关子集,冻结的子代理使用这些工具构建推理链并生成偏好判断。通过比较预测结果与真实偏好,并分析推理过程中的成功与失败,调度器可自动优化工具库,无需额外人工标注。仅使用EditReward数据的0.05%,RewardHarness在图像编辑评估基准上实现47.4%平均准确率,超越GPT-5达5.3个百分点。将其作为奖励信号用于GRPO微调时,强化学习模型在ImgEdit-Bench上得分达3.52。
原文摘要 · Abstract (English)
Evaluating instruction-guided image edits requires rewards that reflect subtle human preferences, yet current reward models typically depend on large-scale preference annotation and additional model training. This creates a data-efficiency gap: humans can often infer the target evaluation criteria from only a few examples, while models are usually trained on hundreds of thousands of comparisons. We present RewardHarness, a self-evolving agentic reward framework that reframes reward modeling as context evolution rather than weight optimization. Instead of learning from large-scale annotations, RewardHarness aligns with human preferences by iteratively evolving a library of tools and skills from as few as 100 preference demonstrations. Given a source image, candidate edited images, and an editing instruction, an Orchestrator selects the most relevant subset of tools and skills from the maintained library, and a frozen Sub-Agent uses them to construct a reasoning chain that produces a preference judgment. By comparing predicted judgments with ground-truth preferences and analyzing successes and failures in the reasoning process, the Orchestrator automatically refines its library of tools and skills without additional human annotation. Using only 0.05% of the EditReward preference data, RewardHarness achieves 47.4% average accuracy on image-editing evaluation benchmarks, surpassing GPT-5 by 5.3 points. When used as a reward signal for GRPO fine-tuning, RL-tuned models achieve 3.52 on ImgEdit-Bench. Project page: https://rewardharness.com.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。