为大模型上下文学习设计测试框架，评估示例质量。
MILE: A Mutation Testing Framework of In-Context Learning Systems
- 提出专用于上下文演示的变异算子与评分机制。
- 实验证明可有效衡量测试用例的可靠性和质量。
- 适合研究大模型推理鲁棒性与提示工程的学者。
上下文学习（ICL）在大语言模型应用中取得显著进展，仅通过少量输入输出示例即可在推理阶段完成新任务学习，无需修改模型参数。这种神秘能力引发了对示范格式与优化的广泛关注，但依然存在黑箱机制和示例选择敏感等问题。受机器学习系统测试技术启发，本文提出一种针对ICL系统的变异测试框架，用于评估测试数据的质量与有效性。首先设计了多种专用于ICL示范的变异算子，并构建相应的变异评分机制。通过大量实验验证了该框架在评估ICL测试集可靠性与质量方面的有效性。代码已开源于https://github.com/weizeming/MILE。
原文摘要 · Abstract (English)
In-context Learning (ICL) has achieved notable success in the applications of large language models (LLMs). By adding only a few input-output pairs that demonstrate a new task, the LLM can efficiently learn the task during inference without modifying the model parameters. Such mysterious ability of LLMs has attracted great research interests in understanding, formatting, and improving the in-context demonstrations, while still suffering from drawbacks like black-box mechanisms and sensitivity against the selection of examples. In this work, inspired by the foundations of adopting testing techniques in machine learning (ML) systems, we propose a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems. First, we propose several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets. With comprehensive experiments, we showcase the effectiveness of our framework in evaluating the reliability and quality of ICL test suites. Our code is available at https://github.com/weizeming/MILE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。