针对大模型对话系统多轮测试难题，提出自动化检测方法MORTAR，显著提升漏洞发现效率。
MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems
- 基于对话级扰动和元测试关系自动生成多轮测试用例
- 相比单轮测试，每用例发现漏洞数量提升150%以上
- 无需大模型判别器，适合资源有限的开发团队使用
随着基于大语言模型的对话系统在日常生活中广泛应用，质量保障变得愈发重要。尽管已有研究成功在单轮测试场景中识别异常行为，但真实场景下多轮交互更为常见，而针对此类交互的测试方法仍研究不足，主要受限于多轮测试中的测试判据问题。本文提出MORTAR，一种面向多轮对话系统的元测试方法，通过形式化多轮测试流程，自动生成包含多轮扰动与元测试关系（MRs）的问答测试用例，并采用自动化MR匹配机制提升灵活性与效率。该方法完全自动化，不依赖大模型判别器。在测试六款主流大语言模型对话系统时，MORTAR相较单轮元测试基线，每测试用例发现漏洞数量提升超过150%，且所揭示漏洞在多样性、精确性和唯一性上更优。MORTAR有望推动更多多轮测试方法发展，帮助开发者在资源受限条件下更全面评估对话系统性能。
原文摘要 · Abstract (English)
With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in single-turn testing scenarios. However, multi-turn interaction is the common real-world usage of dialogue systems, yet testing methods for such interactions remain underexplored. This is largely due to the oracle problem in multi-turn testing, which continues to pose a significant challenge for dialogue system developers and researchers. In this paper, we propose MORTAR, a metamorphic multi-turn dialogue testing approach, which mitigates the test oracle problem in testing LLM-based dialogue systems. MORTAR formalises the multi-turn testing for dialogue systems, and automates the generation of question-answer dialogue test cases with multiple dialogue-level perturbations and metamorphic relations (MRs). The automated MR matching mechanism allows MORTAR more flexibility and efficiency in metamorphic testing. The proposed approach is fully automated without reliance on LLM judges. In testing six popular LLM-based dialogue systems, MORTAR reaches significantly better effectiveness with over 150\% more bugs revealed per test case when compared to the single-turn metamorphic testing baseline. Regarding the quality of bugs, MORTAR reveals higher-quality bugs in terms of diversity, precision and uniqueness. MORTAR is expected to inspire more multi-turn testing approaches, and assist developers in evaluating the dialogue system performance more comprehensively with constrained test resources and budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。