分析57个评估框架，揭示其工程痛点与优化方向。
Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
- 构建五阶段评估框架模型，系统化梳理流程
- 41.4%问题集中在模型/数据/评分集成阶段
- 功能缺失、文档不足、输入校验缺失是主要根源
评估框架是管理模型调用、数据加载、指标计算和结果报告的软件系统。尽管在机器学习基础设施中至关重要，其运行挑战和工程问题却长期未受重视。我们对57个实际使用的评估框架进行了实证研究，提出了一个五阶段框架模型，并将16,560个问题按工作流阶段和根本原因分类。超过40%的问题集中在规范阶段（Specification），该阶段涉及外部模型、数据集和评分裁判的集成。最常见根本原因为功能未实现（24.3%）、文档缺失（20.3%）和输入验证缺失（17.2%），三者合计占已分类问题的61.7%，涵盖现有功能缺陷与阻碍工作流的能力建设缺口。不同阶段问题根源各异：环境不兼容和外部依赖失效占配置问题的36.2%；算法错误（25.9%）和验证漏洞（22.5%）主导评估阶段问题。这些发现为将评估工程视为独立软件工程议题奠定了实证基础。
原文摘要 · Abstract (English)
Evaluation harnesses are software systems that orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting. Despite their critical role in machine learning infrastructure, their operational challenges and engineering concerns have received limited attention so far. We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause. Most harness operational challenges concentrate in the Specification stage (41.4% of issues), where harnesses integrate external models, datasets, and scoring judges. The three most frequent root causes of operational challenges are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), which together account for 61.7% of classified issues, spanning both defects in existing functionality and capability gaps that block intended workflows. Root causes also vary by workflow stage: environment incompatibility and external dependency breakage account for 36.2% of provisioning issues, whereas algorithmic error (25.9%) and validation gap (22.5%) dominate assessment issues. Together, these contributions establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。