给大模型生成的测试断言分类，关键看它依据什么判断正确性。
LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review
- 按断言权威来源分类，区分规范依赖与模型训练所得
- 超半数研究无原始规范即可做出判断，但可信度难验证
- 强调应追问断言依据，适合关注测试可信度的研究者
大型语言模型（LLMs）越来越多地用于判断软件是否正确，或作为测试断言生成器或执行者。然而，两个看似相同的断言可能基于不同基础：一个来自书面规范，另一个仅源于模型训练所学。现有综述多按形式或技术分类，很少关注其判决可信度的根源——权威来源。本研究遵循PRISMA 2020指南，筛选2,436条记录至54项纳入研究，经引文追溯（雪球法）扩展至83项。从权威来源、断言形式、判定机制三维度分析。结果显示，超过一半研究在无任何规范的情况下做出结论。这使其能应用于无规范代码，但也使质疑时缺乏支撑。权威来源与判定机制并不重合，如“大模型作裁判”仅描述输出方式，不说明可信理由。多数研究以与已知断言相似度评估质量，而非能否捕获注入缺陷。因此，对任一大模型断言，首要问题是：其判决依据为何？研究协议、检索查询和每篇编码表均已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly decide whether software behaves correctly, either by writing a test oracle or by acting as one. Yet two oracles can look identical and rest on different ground: one assertion encodes a written specification, another only what the model learned in training. Prior secondary studies sort oracles by form or by technique, rarely by the property that governs how far a verdict can be trusted: where its authority comes from. This systematic literature review, reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, screens 2,436 records to 54 included studies, extended by citation searching (snowballing) to 83 in total. We read the corpus along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. Just over half of the corpus reaches a verdict with no specification at all. That is what lets these oracles work on code with no specification to consult, and what leaves a challenged verdict with less to fall back on. Source and mechanism cross-cut rather than coincide, so a label such as LLM-as-a-judge names how a verdict is produced, not why it should be trusted. Oracle quality is most often judged by resemblance to a known oracle rather than by whether injected faults are caught. The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict. The protocol, search query, and per-study coding sheet are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。