用成对比较替代奖励信号,让智能体自动进化无需人工标注。
Reward-Free Evolving Agents via Pairwise Validator

- 用冻结的LLM做成对判断,决定新旧智能体哪个更好。
- 在多个任务中达到或超过有奖励信号的基线表现。
- 无需训练和标注,适合缺乏奖励设计资源的研究者。
一种自演化智能体循环反复提出改进后的智能体(提示模板或程序),并根据每轮的质量信号决定是否采纳变更。设计该信号通常是项目中最昂贵的部分:可靠的标量奖励需要领域知识和标注数据,而这些数据的收集成本与任务本身相当。我们提出将接受/拒绝门中的标量奖励替换为成对验证器:一个冻结的大型语言模型,给定父代与子代候选,返回二元判断以确定哪个更优。由于对比性质,成对判断通常比绝对评分更易获得且更稳定,从而减少了对严格尺度校准的需求。验证器本身无需训练。我们将该验证器集成到三个已发表的自演化引擎(GEPA、ADRS、ShinkaEvolve)中,报告两种变体:适应性聚焦(保留原有验证集父代选择)与软埃洛(由验证器判断驱动父代选择,使验证集奖励也下降)。在多个智能体和两类生成物(提示与代码)上,我们的方法在多数评估设置中匹配或超越全奖励基线,且跨验证器家族切换后仍保持一致性能。因此,成对判断门可作为每步奖励设计的即插即用替代方案,在不增加标注成本的前提下实现竞争性任务精度。
原文摘要 · Abstract (English)
A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal. Designing that signal is often the costly part of the project: a reliable scalar reward requires domain expertise and labeled examples that are themselves as expensive to assemble as the agent's underlying task. We propose replacing the scalar at the accept/reject gate with a pairwise validator: a frozen LLM that, given the parent and child candidate, returns a binary verdict on which is better. Pairwise judgment is generally easier and more stable than absolute scoring, due to its contrastive nature, which mitigates the need for strict scale calibration. The validator also requires no training of its own. We integrate the validator into three published self-evolving engines (GEPA, ADRS, ShinkaEvolve) and report two flavors: Adaptive Focus, which retains the engine's existing val-set parent selection, and Soft Elo, which lets the validator's verdicts drive parent selection so that val-set rewards drop as well. Across multiple agents and two artifact substrates (prompt and code), our method matches or exceeds the full-reward baseline on the majority of settings we evaluate, and the pattern survives a cross-family validator swap. The pairwise gate is thus a drop-in replacement for per-step reward design at competitive task accuracy without the labeling cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。