让评估器自我进化,从错误中学习并自动写出高效代码检测规则。
Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots
- 用程序验证中的反例引导抽象精化思想,从模型错误中生成评估算子
- 在未见任务上减少15.4%的评估空白,仅用55行代码实现手写规则四分之一的标记量
- 适合需要自动化评估但缺乏评分标准的场景,如报告生成与代码生成
智能体在有可靠自动评估指标时进步迅速,而缺乏指标则停滞不前。然而,在最需要智能体的应用(如报告生成)中,往往无人知晓如何评分。能否让评估指标自我生成?虽然描述优质回答困难,但指出具体错误相对容易。本文提出一个由小型Python算子组成的评估池,每个算子负责识别一种特定缺陷或选择不判断,并通过投票机制决定结果。直接让大模型生成这些算子失败:183个候选仅产生96种不同行为,局限于巨大空间中的狭窄区域。为此,作者借鉴程序验证中的反例引导抽象精化(CEGAR)思想,将算子池视为抽象,搜索出一对“冲突样本”——即两个表现相同的答案,一正确一错误。这一对样本作为生成新算子的指令,而非提示词。当冲突无法被现有算子区分时,系统扩大算子可读取的信息范围,而非重新采样。在MBPP+和HumanEval+(含隐藏单元测试的沙盒环境)上,该方法自动生成了一个55行的算子,在428个未见任务中缩小了15.4%的评估差距(提升+0.0065,p=0.0010),仅需手写最优算子四分之一的标记量。在未见过的基准上,其效果与手写算子完全匹配,覆盖约三分之一的标记。六次实验中有六次成功生成有效算子,全部提升泛化性能;而15个手写算子合并后反而降低准确率。相比之下,使用大模型作为裁判虽能达到相似性能,但需为每个候选者调用一次模型,成本高昂且不可持续。
原文摘要 · Abstract (English)
Agents improve quickly against a reliable automatic metric and stall without one, and the applications that need them most, report generation among them, are the ones nobody knows how to score. Can the metric write itself? Saying what makes an answer good is hard; pointing at something wrong with one is easier, so the metric we evolve is a pool of small Python operators that each flag a candidate for one named defect, or abstain, and vote. Asking a model for operators directly does not work: 183 candidates realise only 96 distinct behaviours, from one narrow region of an enormous space. EvalCEGAR instead borrows counterexample-guided abstraction refinement from program verification. It reads the pool as an abstraction and searches for a collision, two answers the operators score identically, one correct and one not. That pair, not a prompt, is the authoring request, and when a collision defeats every attempt the loop widens what an operator may read rather than resampling. On MBPP+ and HumanEval+, a sandbox whose hidden unit tests give exact ground truth, the loop writes a 55-line operator that closes 15.4% of the gap between flagging nothing and a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of our best hand-written operator's flags. On the benchmark it never saw it matches that operator's effect exactly on a third of the flags. Six of eight runs admit such an operator and all six help out of sample; our 15 hand-written operators applied together as one filter lose accuracy. An LLM judge on the same information ties that delta on a nearly disjoint set of candidates, and charges a model call per candidate forever where the operator charges none.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。