根据任务自动生成评价标准，让大模型代理评估更准确可靠。
AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning
- 用大模型从任务描述生成定制化评价维度
- 在多个数据集上实现0.79的人类相关性（+0.15）
- 适合需要精准评估智能体表现的研究者
评估大模型代理轨迹本质上是任务特定的：代码调试代理应以正确性和错误处理为评判标准，而非流畅性或安全性。然而主流方法——使用固定评价维度的大模型作为评判者——对所有任务采用相同标准，导致系统性误评。我们提出AdaRubric框架，(i)通过大模型从任务描述中自适应生成任务特定的评价标准，(ii)分步评估代理轨迹并进行置信度加权的维度评分，(iii)生成密集奖励信号用于偏好学习。三种可组合的过滤策略，包括新提出的维度感知过滤（DimensionAwareFilter），可证明避免维度级质量掩盖问题，生成高质量的DPO偏好对。在WebArena、ToolBench和AgentBench上，AdaRubric达到0.79的人类相关性（较最强基线提升0.15），可靠性高（Krippendorff's alpha = 0.83）。基于AdaRubric生成的偏好对训练的DPO模型，在任务成功率上比最佳基线提升6.8%-8.5%。该方法还能零样本泛化至未见领域（SWE-bench）并直接扩展到多模态代理（VisualWebArena, OSWorld），无需修改。代码已开源：github.com/alphadl/AdaRubrics。
原文摘要 · Abstract (English)
Evaluating LLM agent trajectories is fundamentally task-specific: a code-debugging agent should be judged on Correctness and Error Handling, not on Fluency or Safety. Yet the dominant paradigm -- LLM-as-Judge with a fixed rubric -- applies the same static dimensions regardless of task, producing systematic mis-evaluation. We present AdaRubric, a framework that (i) adaptively generates task-specific evaluation rubrics from task descriptions via LLM, (ii) evaluates agent trajectories step-by-step with confidence-weighted, per-dimension scoring, and (iii) produces dense reward signals for preference learning. Three composable filtering strategies, including the novel DimensionAwareFilter that provably prevents dimension-level quality masking, yield high-quality DPO preference pairs. On WebArena, ToolBench, and AgentBench, AdaRubric achieves Pearson r = 0.79 human correlation (+0.15 over the strongest baseline), with strong reliability (Krippendorff's alpha = 0.83). DPO models trained on AdaRubric-generated pairs improve task success by +6.8-8.5% over the best baseline. AdaRubric also generalises zero-shot to unseen domains (SWE-bench) and extends to multimodal agents (VisualWebArena, OSWorld) without modification. Our code is available at: github.com/alphadl/AdaRubrics
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。