自动生成多样提示词，让大模型评估更可靠。
PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation
- 模块化设计，可精准控制每个提示组件的改动
- 支持多任务、多基准，无需手动构造提示变体
- 开源工具包含API与网页界面，开箱即用
单一提示词评估大模型表现不可靠，微小变化即引发性能波动。而人工生成足够多的提示变体以实现稳健评估又极具挑战性，限制了其实际应用。为此，我们提出 PromptSuite，一个任务无关的自动提示生成框架。该框架具备高度灵活性，可直接应用于多种任务与基准测试。其采用模块化提示设计，支持对每个组件进行受控扰动，并具备可扩展性，可轻松添加新组件与扰动类型。通过一系列案例研究，我们验证了 PromptSuite 能有效生成有意义的提示变体，助力更科学的评估实践。所有资源，包括 Python API、源码、用户友好的网页界面及演示视频，均已公开：https://eliyahabba.github.io/PromptSuite/。
原文摘要 · Abstract (English)
Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for a more robust multi-prompt evaluation is challenging, limiting its adoption in practice. To address this, we introduce PromptSuite, a framework that enables the automatic generation of various prompts. PromptSuite is flexible - working out of the box on a wide range of tasks and benchmarks. It follows a modular prompt design, allowing controlled perturbations to each component, and is extensible, supporting the addition of new components and perturbation types. Through a series of case studies, we show that PromptSuite provides meaningful variations to support strong evaluation practices. All resources, including the Python API, source code, user-friendly web interface, and demonstration video, are available at: https://eliyahabba.github.io/PromptSuite/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。