用心理测评思路预测代码智能体在具体任务中的表现
Agent psychometrics: Task-level performance prediction in agentic coding benchmarks
- 将题目特征与智能体能力分解为大模型和框架两部分
- 在未见过的基准上准确预测任务通过率
- 适合想优化评测难度的基准设计者使用
随着大模型编程从单步生成转向多步工具交互，理解哪些任务会难倒智能体变得愈发复杂。当前普遍采用整体通过率衡量性能，但单一数值掩盖了基准中任务的多样性。本文提出一种面向智能体编程场景的任务级表现预测框架，将项目反应理论（IRT）与任务特征（如问题描述、代码库上下文、解决方案、测试用例）结合，并首次将智能体能力分解为大模型能力与框架能力。该参数化方法可跨异构排行榜聚合评估数据，准确预测未见基准及未见大模型-框架组合下的任务表现。本方法对基准设计者具有实用价值，可无需昂贵的智能体评估即可校准新任务难度。
原文摘要 · Abstract (English)
As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult. This is compounded by current practice: agent performance is typically measured by aggregate pass rates on benchmarks, but single-number metrics obscure the diversity of tasks within a benchmark. We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime. Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases, and introduces a novel decomposition of agent ability into LLM and scaffold ability components. This parameterization enables us to aggregate evaluation data across heterogeneous leaderboards and accurately predict task-level performance for unseen benchmarks, as well as unseen LLM-scaffold combinations. Our methods have practical utility for benchmark designers, who can better calibrate the difficulty of their new tasks without running computationally expensive agent evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。