arXiv:2509.17314cs.SEcs.LG2025-09被引 5

用模型隐状态预判输入难易,提前发现高风险测试用例。

Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

  • 基于GMM从隐藏状态直接评估输入难度,无需生成输出。
  • 仅标注5.4%的输入即可预测失败,ROC-AUC达0.716。
  • 可迁移至闭源模型,显著提升测试用例筛选效率。

软件越来越多地依赖大语言模型(LLMs)的涌现能力,涵盖自然语言理解、程序分析与生成等任务。然而,针对特定任务的测试仍面临困难且成本高昂:许多提示缺乏真实标签,只能依赖人工判断;现有测试充分性度量通常基于输出不确定性,仅在完整推理后可用。关键挑战在于,在生成任何输出前,评估输入对任务的有用性。本文提出Clotho,一种任务特定的、推理前的测试充分性度量方法,通过分析LLM隐藏状态直接估计输入难度。对于某一任务的大规模无标签输入池,Clotho利用高斯混合模型(GMM)自适应采样最具信息量的样本供人工标注。基于该参考集,GMM可对未见输入按失败概率排序。在八个基准任务和三种开源权重LLM上的实证评估显示,Clotho在仅标注平均5.4%输入的情况下,预测失败的ROC-AUC达0.716,且无需生成输出,大幅降低模型执行成本。与事后生成的度量相比,两者具有互补性。更重要的是,我们证明了由开源模型学习的充分性评分可有效迁移至专有模型,显著提升测试输入优先级效果:相比随机排序,每100个输入中失败案例数从18.7提高至42.5。

原文摘要 · Abstract (English)

Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation. Yet testing them on specific tasks remains difficult and costly: many prompts lack ground truths, forcing reliance on human judgments, while existing test adequacy measures typically rely on output uncertainty and thus are only available after full inference. A key challenge is to assess how useful a test input is in a way that reflects the demands of the task, ideally before even generating any output. We introduce Clotho, a task-specific, pre-generation test adequacy measure that estimates input difficulty directly from LLM hidden states. Given a large pool of unlabelled inputs for a specific task, Clotho uses a Gaussian Mixture Model (GMM) to adaptively sample the most informative cases for human labelling. Based on this reference set the GMM can then rank unseen inputs by their likelihood of failure. In our empirical evaluation across eight benchmark tasks and three open-weight LLMs, Clotho can predict failures with a ROC-AUC of 0.716, after labelling reference sets that are on average only 5.4% of inputs. It does so without generating any outputs, thereby significantly reducing LLM execution costs compared to output-based uncertainty or confidence measures. Comparison of Clotho and these post-generation adequacy measures shows that the two approaches complement each other. Crucially, we show that adequacy scores learnt from open-weight LLMs transfer effectively to proprietary models, extending the applicability of the approach. When prioritising test inputs for proprietary models, Clotho increases the average number of failing inputs from 18.7 to 42.5 out of 100, compared to random prioritisation.

测试评估大模型预生成迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。