用自动检测方法发现大模型输出错误，无需人工标注数据。
LLMORPH: Automated Metamorphic Testing of Large Language Models
- 基于变异测试生成新输入，检验模型输出一致性。
- 在4个基准上测试3个大模型，触发超56万次测试执行。
- 适合研究者和开发者评估大模型鲁棒性，可灵活扩展。
自动化测试对评估和提升大语言模型（LLMs）的可靠性至关重要，但缺乏自动化的验证机制仍是关键挑战。本文提出 LLMORPH，一个专为执行自然语言处理任务的大型语言模型设计的自动化测试工具，利用变异测试（Metamorphic Testing, MT）在不依赖人工标注数据的情况下发现模型缺陷。MT通过变异关系（MRs）从原始输入生成后续输入，从而检测模型输出中的不一致现象。我们详细阐述了 LLMORPH 的设计、实现与实际应用，展示了其可轻松扩展至任意大模型、自然语言处理任务及变异关系组合。在评估中，我们在四个 NLP 基准上应用 36 种 MRs，测试了三种前沿大模型：GPT-4、LLAMA3 与 HERMES 2，共完成超过 561,000 次测试执行。结果表明，LLMORPH 能有效自动暴露模型输出中的不一致性。
原文摘要 · Abstract (English)
Automated testing is essential for evaluating and improving the reliability of Large Language Models (LLMs), yet the lack of automated oracles for verifying output correctness remains a key challenge. We present LLMORPH, an automated testing tool specifically designed for LLMs performing NLP tasks, which leverages Metamorphic Testing (MT) to uncover faulty behaviors without relying on human-labeled data. MT uses Metamorphic Relations (MRs) to generate follow-up inputs from source test input, enabling detection of inconsistencies in model outputs without the need of expensive labelled data. LLMORPH is aimed at researchers and developers who want to evaluate the robustness of LLM-based NLP systems. In this paper, we detail the design, implementation, and practical usage of LLMORPH, demonstrating how it can be easily extended to any LLM, NLP task, and set of MRs. In our evaluation, we applied 36 MRs across four NLP benchmarks, testing three state-of-the-art LLMs: GPT-4, LLAMA3, and HERMES 2. This produced over 561,000 test executions. Results demonstrate LLMORPH's effectiveness in automatically exposing inconsistencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。