揭示了提示词难找的根源，发现最优提示常违背直觉。
Why is prompting hard? Understanding prompts on binary sequence predictors
- 从预训练分布出发，分析最优提示的生成机制。
- 即使穷举搜索，也难找到实用神经模型的最佳提示。
- 实证发现常用演示式提示反而次优，适合研究提示机理者阅读。
前沿模型可通过提示完成多种任务，但寻找有效提示并不容易，也难以理解为何某些提示表现优异。本文将提示问题视为在近似最优序列预测器上寻找最佳条件序列。通过一系列受控实验，我们发现：若缺乏预训练分布信息，最优条件序列往往难以理解；即便采用穷举搜索，可靠识别实际神经预测器的最优提示也出人意料地困难。常见的提示方法（如使用目标任务的示例）可能显著次优。基于相同实证框架，我们进一步分析前沿模型上的最优提示，发现其模式与二进制序列示例及先前研究结果一致。该工作首次从统计与实证角度切入，为理解最优提示提供了新视角，补充了对前沿模型的研究。
原文摘要 · Abstract (English)
Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal conditioning sequences can be better understood given the pretraining distribution, which is not usually available. Even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Popular prompting methods, such as using demonstrations from the targeted task, can be surprisingly suboptimal. Using the same empirical framework, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings. Taken together, this work takes an initial step towards understanding optimal prompts, from a statistical and empirical perspective that complements research on frontier models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。