构建可复现的生成式AI测试框架，验证模型更新与提示词的效果。
Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets
- 设计轻量级基准GPR-bench，结合双语数据集与自动化评分
- 新模型正确率略有提升但差异不显著，提示词优化使简洁性提升12.37个百分点
- 适合关注AI系统可靠性、需定期测试的开发者和研究者
生成式AI系统的可复现性与可靠性仍是核心挑战，其行为可能随模型更新或提示词修改而漂移。本文提出GPR-bench，一个轻量且可扩展的基准，用于通用场景下的回归测试。该框架整合了一个开放的双语（英文与日文）数据集，涵盖八类任务（如文本生成、代码生成、信息检索）及每类任务下10种场景，共80个测试用例（每语言）。配套的自动化评估流程采用“大模型作为裁判”对正确性与简洁性进行评分。在gpt-4o-mini、o3-mini和o4-mini三个近期模型版本，以及默认提示与简洁写作指令两种配置下的实验显示，模型质量存在异质性。结果表明，新模型总体上提升正确率，但差异微小且无统计显著性，说明当前基准可能不足以区分最新模型版本。相比之下，简洁写作指令显著提升简洁性（+12.37个百分点，Mann-Whitney U检验：p < 0.001，效应量r = 0.2995），同时仅轻微降低准确率（-1.7个百分点），凸显提示工程的有效性。GPR-bench已开源（MIT许可），降低了可复现性监控门槛，并为社区扩展提供基础，同时也引发关于快速演进语言模型评测基准设计的重要思考。
原文摘要 · Abstract (English)
Reproducibility and reliability remain pressing challenges for generative AI systems whose behavior can drift with each model update or prompt revision. We introduce GPR-bench, a lightweight, extensible benchmark that operationalizes regression testing for general purpose use cases. GPR-bench couples an open, bilingual (English and Japanese) dataset covering eight task categories (e.g., text generation, code generation, and information retrieval) and 10 scenarios in each task categories (80 total test cases for each language) with an automated evaluation pipeline that employs "LLM-as-a-Judge" scoring of correctness and conciseness. Experiments across three recent model versions - gpt-4o-mini, o3-mini, and o4-mini - and two prompt configurations (default versus concise-writing instruction) reveal heterogeneous quality. Our results show that newer models generally improve correctness, but the differences are modest and not statistically significant, suggesting that GPR-bench may not be sufficiently challenging to differentiate between recent model versions. In contrast, the concise-writing instruction significantly enhances conciseness (+12.37 pp, Mann-Whitney U test: p < 0.001, effect size r = 0.2995) with minimal degradations on accuracy (-1.7 pp), demonstrating the effectiveness of prompt engineering. Released under the MIT License, GPR- bench lowers the barrier to initiating reproducibility monitoring and provides a foundation for community-driven extensions, while also raising important considerations about benchmark design for rapidly evolving language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。