RETAIN工具帮助开发者在大模型迁移时自动发现性能退化问题。
RETAIN: Interactive Tool for Regression Testing Guided LLM Migration
- 设计交互式界面与错误发现模块，定位模型输出差异
- 相比手动测试，可多发现2倍错误，实验提示75%更多
- 适合需要频繁迁移大模型的开发团队使用
大型语言模型（LLMs）正被广泛集成到各类应用中。随着模型快速迭代，开发者持续优化应用成为可能，但模型迁移也可能导致性能退化。尽管已有若干交互式工具简化提示工程，但针对大模型迁移中的回归测试需求仍缺乏专门解决方案。为此，我们提出RETAIN（REgression Testing guided LLM migrAtIoN），一个专为大模型迁移回归测试设计的工具。RETAIN包含两个核心组件：面向迁移测试需求的交互界面，以及支持理解模型行为差异的错误发现模块。该模块能生成输出差异的文本描述，提供可操作的提示优化建议。自动评估与用户实证研究显示，相较于人工评估，使用RETAIN的参与者可发现两倍以上的错误，实验的提示数量增加75%，且在相同时间内指标得分提升12%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly integrated into diverse applications. The rapid evolution of LLMs presents opportunities for developers to enhance applications continuously. However, this constant adaptation can also lead to performance regressions during model migrations. While several interactive tools have been proposed to streamline the complexity of prompt engineering, few address the specific requirements of regression testing for LLM Migrations. To bridge this gap, we introduce RETAIN (REgression Testing guided LLM migrAtIoN), a tool designed explicitly for regression testing in LLM Migrations. RETAIN comprises two key components: an interactive interface tailored to regression testing needs during LLM migrations, and an error discovery module that facilitates understanding of differences in model behaviors. The error discovery module generates textual descriptions of various errors or differences between model outputs, providing actionable insights for prompt refinement. Our automatic evaluation and empirical user studies demonstrate that RETAIN, when compared to manual evaluation, enabled participants to identify twice as many errors, facilitated experimentation with 75% more prompts, and achieves 12% higher metric scores in a given time frame.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。