评测编程智能体在编译框架中的代码生成能力，真实测试其功能实现与兼容性。
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
- 基于PyTorch扩展的张量框架，设计199个真实功能添加与重构任务
- 通过自动运行测试套件评估，最强模型通过率64.8%，最弱仅22.1%
- 任务涵盖稀疏格式、优化通路、运行时组件等，适合评估代码生成可靠性
仓库级编程基准面临任务难度与评估可靠性之间的权衡：挑战前沿模型的任务常涉及大型代码库且测试覆盖率不全，而人工评审又无法扩展。我们提出TensorBench，一个针对开源编译型张量框架（扩展PyTorch以原生支持稠密与稀疏张量）的199个功能添加与重构任务基准。任务覆盖新稀疏格式、稠密优化通路、IR变换、调度器修改、运行时组件及高层数值算子。每个任务通过应用智能体提交的补丁并运行框架自带测试套件进行评分，包括已有随机回归测试和智能体新增的测试。对于功能添加任务，通过标准为：补丁保留原有行为，并满足智能体新增的特性校验。我们评估了七个跨越三个前沿模型家族及一个开源权重模型的编程智能体。在该标准下，通过率从最强的64.8%到最弱的22.1%不等。不同智能体通过不同子集任务，成对的Cohen's κ值介于-0.07至0.43之间，最强两模型间κ=0.05。
原文摘要 · Abstract (English)
Repository-level coding benchmarks face a trade-off between task difficulty and evaluation reliability: tasks that challenge frontier models often involve large codebases with incomplete test coverage, while human review does not scale. We introduce TensorBench, a benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework that extends PyTorch with first-class support for dense and sparse tensors. Tasks cover new sparse formats, dense optimization passes, IR transformations, scheduler changes, runtime components, and high-level numerical operators. TensorBench grades each run by applying the agent's patch and running the framework's test suite, which includes the pre-existing randomized regression tests and any tests the agent adds. For feature-addition tasks, a pass means that the patched repository preserves the tested pre-existing behavior and satisfies the agent-added checks for the requested feature. We evaluate seven coding agents spanning three frontier model families and one open-weight model. Pass rates under this criterion range from $64.8\%$ for the strongest agent to $22.1\%$ for the weakest. Agents pass different subsets of tasks: pairwise Cohen's $κ$ ranges from $-0.07$ to $0.43$, with $κ= 0.05$ for the two strongest agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。