提出新方法评估测试题价值，发现部分题目可优化
Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation
- 将评测视为多方博弈，从福利、可改进性、方差三维度评估题目
- 在OLMES数据集上发现部分题目在工人福利下表现劣于其他题目
- 开源工具可审计评测题质量，适合评测设计者和研究者使用
AI评测存在诸多缺陷，如题目污染、饱和与概念模糊。现有方法普遍采用平均法汇总题目得分，隐含假设所有题目价值相同。本文将评测建模为多任务委托-代理问题，指出评测整体福利损失由三项题目级基础特征共同决定：与规范福利目标的对齐度、边际可改进性、性能方差。据此构建审计框架，沿三条轴线对题目排序，并在OLMES题目上应用：以WORKBank衡量福利，EvoLM 4B套件评估可改进性，PolyPythias 410M面板分析方差。结果发现，在强调工人福利的设定下，部分题目属于帕累托劣于其他题目的情况。所有代码已公开于https://github.com/stair-lab/principal-agent-benchmarks。
原文摘要 · Abstract (English)
AI benchmarks have well-documented limitations, with prior work examining contamination, saturation, and construct underspecification. Aggregation has received far less attention: benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives: alignment with normative welfare priorities, marginal improvability, and performance variance. We translate the theory into an audit framework that ranks items along each of these three axes, and apply it to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance. The framework surfaces items that are Pareto-inferior within OLMES subject to a pro-worker welfare operationalization. All code is available at https://github.com/stair-lab/principal-agent-benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。