构建新基准评估智能体执行过程中的技能判断能力。
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution
- 设计包含681个真实轨迹的跨领域评测集，结合任务技能与可验证证据。
- 引入可复用的判断技能（JudgeSkill），使智能体裁判准确率提升14.8个百分点。
- 适合研究智能体自主验证、自动化评测及强化学习中可靠性评估的学者。
大模型智能体通过工具调用和环境交互执行长时序任务，评价方式正从最终输出评分转向对完整执行过程的验证。对于技能增强型智能体，验证还需依赖任务期间嵌入的程序性知识，以判断应检查哪些证据及何种失败为关键。然而现有评测基准常仅暴露最终结果或静态轨迹，很少将任务技能与可直接查验的实体和环境结合。为此，我们提出SkillTV-Bench，一个涵盖11个领域50个任务共681个真实智能体轨迹的基准，用于评估基于大模型或智能体的裁判在技能感知下的轨迹验证能力。此外，我们提出SkillTV-Evolve，将验证知识外化为可复用的JudgeSkill，指导智能体裁判规划针对性检查并给出基于证据的判决。在独立开发集上，自动演化循环通过误判案例进一步优化该技能。在SkillTV-Bench上，优化后的技能使同一智能体裁判准确率提升14.8个百分点；在离线回放池选择中，成功率达22.9%（单次回放）升至45.5%（十次回放）。代码与数据已开源。
原文摘要 · Abstract (English)
LLM agents increasingly execute long-horizon tasks through tool use and environment interaction, shifting evaluation from final-response scoring to verification of complete executions. For skill-augmented agents, verification additionally requires the procedural knowledge encoded in task-time skills, because this knowledge indicates what evidence to inspect and which failures are task-critical. However, existing judge benchmarks often expose final responses or static trajectories, and rarely combine task-time skills with directly inspectable artifacts and environments. We therefore introduce SkillTV-Bench, a 681-case benchmark of real agent trajectories from 50 tasks across eleven domains, designed to evaluate skill-aware trajectory verification for both LLM-as-a-Judge and Agent-as-a-Judge methods. Additionally, we propose SkillTV-Evolve, which externalizes verification knowledge as a reusable JudgeSkill that guides an agent judge to plan targeted inspections and issue evidence-grounded verdicts. On a disjoint development pool, an automated evolution loop further refines the JudgeSkill using misjudged cases. On SkillTV-Bench, the refined skill increases the same agent judge's accuracy by 14.8 percentage points. In offline rollout-pool selection, it increases selected-trajectory success from 22.9% with one rollout to 45.5% with ten rollouts. The code and data are available at https://github.com/HanZhi306/SkillTV-Bench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。