用智能生成少量示例，让大模型在数据少时也能快速高效地理解任务。
PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data
- 通过蒙特卡洛谢帕利估计自动筛选最优示例，迭代优化提示词。
- 在有限算力下，文本简化和GSM8K任务上超越现有方法，分类与摘要也表现优异。
- 适合资源受限场景下的快速提示工程，尤其适合小样本任务应用。
大型语言模型对提示设计极为敏感，但手工构造有效提示困难且需精心设计少样本示例。本文提出一种快速自动提示构建算法PIAST，通过生成少量少样本示例来增强人工指令。方法基于蒙特卡洛谢帕利估计评估示例效用，迭代替换、删除或保留示例。为加速执行，采用激进子采样和重播缓冲区以提升评估效率。该方法可适应不同计算时间预算：在有限预算下，在文本简化和GSM8K任务上优于现有自动提示方法，并在分类和摘要任务中取得第二名；在适度扩展但依然可控的算力预算下，于分类、简化与GSM8K任务上达到自动提示方法的新最佳性能。结果表明，精心构造的示例比广泛搜索指令更能显著提升快速、数据高效的提示工程效果。代码已公开于https://github.com/Batorskq/PIAST。
原文摘要 · Abstract (English)
LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. On a limited budget, we outperform existing automatic prompting methods on text simplification and GSM8K and obtain second best results on classification and summarization. With an extended, but still modest compute budget we set a new state of the art among automatic prompting methods on classification, simplification and GSM8K. Our results show that carefully constructed examples, rather than exhaustive instruction search, are the dominant lever for fast and data efficient prompt engineering. Our code is available at https://github.com/Batorskq/PIAST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。