用上下文信息指导模型测试，发现更关键的故障。
Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models
- 引入上下文作为先验，引导大模型推测可能的失败点。
- 在多个场景中自动识别出比传统方法更相关、更严重的缺陷。
- 适合关注模型可靠性与安全性研究者使用。
当前主流机器学习模型测试方法仅依赖保留数据计算整体评估指标，或分组评估性能。这类纯数据测试方法假设可用数据是唯一输入，忽略了可指导测试的上下文信息。本文挑战这一默认做法，提出上下文感知测试（CAT），利用上下文作为归纳偏置，引导寻找有意义的模型失效。我们构建首个CAT系统SMART Testing，通过大语言模型生成可能的故障假设，并以自证伪机制在数据上验证。在多种场景的实证评估表明，SMART能自动发现比现有方法更相关、更具影响力的故障，展示了CAT作为新测试范式的潜力。
原文摘要 · Abstract (English)
The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。