用代码自动生成评估指标,无需人工标注就能精准衡量AI系统表现。
Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

- 从源代码中提取评估意图,自动设计每阶段的评估指标。
- 在三个真实场景中覆盖最广、冗余最低,科学问答任务相关性达0.72。
- 适合需要可解释、可验证评估的AI系统开发者与审计人员。
随着智能体大模型系统从原型走向多领域部署,其评估变得愈发重要也愈发困难。问题不仅在于单一指标不可靠,更在于评估目标常被隐含。缺乏对系统预期行为、可能失败方式及关键失效点的明确说明,使得指标选择难以论证、解读或验证。本文提出Litmus,一种零标签系统,通过分析源代码并进行定向提问,自动获取评估意图,并据此构建合理、分阶段的指标组合。我们将其应用于金融账户分组、科学问答和固有风险评估三个真实代码定义的AI流水线,对比AutoMetrics和三种DynamicRubric基线。Litmus在关注点覆盖范围上最广或并列第一,涵盖更多流水线阶段,指标组合近零冗余,在所有三类任务中基于逐行质量标签的验证得分最高——尤其在科学问答任务中显著领先(斯皮尔曼相关系数ρ=0.72,基线均低于0.47),且在审计框架两个维度上与最优结果处于重叠置信区间,尽管指标设计全程未使用任何标签。结果支持从自动指标实现转向自动指标规范:评估前应先明确‘必须测量什么’及‘为何要测’。
原文摘要 · Abstract (English)
As agentic LLM systems move from prototypes to deployment across increasingly diverse domains, evaluating them has become both more important and more difficult. The challenge is not only that individual metrics may be unreliable, but that evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Instead of assuming that the evaluation target is already known, Litmus first identifies what must be measured and why, then converts those answers into constraints for constructing a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined AI pipelines - financial account grouping, scientific QA, and inherent risk assessment - against AutoMetrics and three DynamicRubric baselines. Litmus achieves the broadest or tied-broadest concern coverage, spans more pipeline stages, produces a near-zero-redundancy portfolio, and ranks first in validity against per-row quality labels on all three pipelines - decisively on scientific QA (Spearman $ρ=0.72$ vs. less than $0.47$ for every baseline), and within overlapping confidence intervals in relation to two components of the audit framework despite using no labels during metric design. Our results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。