arXiv:2507.05565cs.SEcs.AI2025-07被引 3

用搜索算法优化大模型测试中的变异关系,提升漏洞发现率并降低计算成本。

Search-based Selection of Metamorphic Relations for Optimized Robustness Testing of Large Language Models

  • 设计基于多目标优化的搜索算法,自动选择最优变异关系组合。
  • MOEA/D算法表现最佳,能高效发现使大模型出错的关键测试用例。
  • 适用于大模型鲁棒性评估,尤其适合追求高覆盖率与低开销的研究者。

评估大语言模型(LLMs)的可信度,尤其是鲁棒性,已成为研究热点。现有基于变异关系(MRs)的测试方法虽广泛使用,但需大量可扩展的MRs,亟需优化选择策略。多数研究仅限于生成单一扰动的测试用例,覆盖范围有限。本文提出一种基于搜索的MR组优化方法,旨在最大化故障检测率并最小化模型执行成本。该方法支持组合扰动,显著拓展测试空间。我们设计了四种搜索算法(Single-GA、NSGA-II、SPEA2、MOEA/D),并引入新型编码方式解决MR选择问题。在两个主流大模型的Text-to-Text任务上,对比实验表明:(1) MOEA/D在优化MR空间方面表现最优;(2) 发现若干“银弹级”MRs,在不同任务中均能有效诱导模型错误。本研究为大模型鲁棒性测试的高效优化提供了新思路。

原文摘要 · Abstract (English)

Assessing the trustworthiness of Large Language Models (LLMs), such as robustness, has garnered significant attention. Recently, metamorphic testing that defines Metamorphic Relations (MRs) has been widely applied to evaluate the robustness of LLM executions. However, the MR-based robustness testing still requires a scalable number of MRs, thereby necessitating the optimization of selecting MRs. Most extant LLM testing studies are limited to automatically generating test cases (i.e., MRs) to enhance failure detection. Additionally, most studies only considered a limited test space of single perturbation MRs in their evaluation of LLMs. In contrast, our paper proposes a search-based approach for optimizing the MR groups to maximize failure detection and minimize the LLM execution cost. Moreover, our approach covers the combinatorial perturbations in MRs, facilitating the expansion of test space in the robustness assessment. We have developed a search process and implemented four search algorithms: Single-GA, NSGA-II, SPEA2, and MOEA/D with novel encoding to solve the MR selection problem in the LLM robustness testing. We conducted comparative experiments on the four search algorithms along with a random search, using two major LLMs with primary Text-to-Text tasks. Our statistical and empirical investigation revealed two key findings: (1) the MOEA/D algorithm performed the best in optimizing the MR space for LLM robustness testing, and (2) we identified silver bullet MRs for the LLM robustness testing, which demonstrated dominant capabilities in confusing LLMs across different Text-to-Text tasks. In LLM robustness assessment, our research sheds light on the fundamental problem for optimized testing and provides insights into search-based solutions.

大模型测试变异测试多目标优化鲁棒性评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。