用变异测试发现大模型的错误行为，无需标注数据
Metamorphic Testing of Large Language Models for Natural Language Processing
- 通过设计输入间关系规则检测大模型输出异常
- 在56万次测试中发现3个主流大模型存在缺陷
- 适合关注大模型可靠性与测试方法的研究者
近年来，利用大语言模型（LLMs）执行自然语言处理（NLP）任务日益普遍。尽管最新LLMs性能卓越，但多项研究显示其常产生错误结果。自动识别这些缺陷对提升LLM有效性至关重要。主要障碍是标注数据稀缺，难以依赖明确的正确性判断（即测试金标准）。变异测试（MT）是一种缓解此问题的方法，核心在于定义相关输入间的输出关系（即变异关系，MR）。MT可在无需显式金标准（如标注数据）的情况下暴露模型缺陷。本文是迄今最全面的针对LLMs的变异测试研究：我们系统回顾文献并收集了191个适用于NLP任务的MR；选取其中36个代表性MR，对三个主流大模型进行实验，共执行约56万次变异测试。结果揭示了MT在检测LLM缺陷方面的潜力与局限。
原文摘要 · Abstract (English)
Using large language models (LLMs) to perform natural language processing (NLP) tasks has become increasingly pervasive in recent times. The versatile nature of LLMs makes them applicable to a wide range of such tasks. While the performance of recent LLMs is generally outstanding, several studies have shown that they can often produce incorrect results. Automatically identifying these faulty behaviors is extremely useful for improving the effectiveness of LLMs. One obstacle to this is the limited availability of labeled datasets, which necessitates an oracle to determine the correctness of LLM behaviors. Metamorphic testing (MT) is a popular testing approach that alleviates this oracle problem. At the core of MT are metamorphic relations (MRs), which define relationships between the outputs of related inputs. MT can expose faulty behaviors without the need for explicit oracles (e.g., labeled datasets). This paper presents the most comprehensive study of MT for LLMs to date. We conducted a literature review and collected 191 MRs for NLP tasks. We implemented a representative subset (36 MRs) to conduct a series of experiments with three popular LLMs, running approximately 560,000 metamorphic tests. The results shed light on the capabilities and opportunities of MT for LLMs, as well as its limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。