arXiv:2503.10084cs.CL2025-03ACL被引 34

揭示提示词设计为何影响大模型推理效果,提供可优化的理论框架。

Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs

  • 提示词如筛选器,从模型隐藏状态中提取关键信息。
  • 最优提示搜索使推理任务性能提升超50%。
  • 适合想系统化提升提示工程的开发者和研究者。

尽管大型语言模型(LLMs)取得了显著成功,但其基于Transformer的架构在处理复杂推理任务时存在固有局限。链式思维(CoT)提示已成为一种实用解决方案,但多数方法依赖单一通用提示(如“逐步思考”),缺乏任务特异性调整。这些方法要求模型自行探索庞大的提示空间以发现有效推理路径。相比之下,部分研究尝试设计任务特定提示,但通常通过试错实现,缺乏理论基础。因此,提示工程仍处于盲目摸索阶段。本文提出一个理论框架,解释为何某些提示有效而其他提示无效。我们证明提示作为选择器,在CoT推理过程中从模型完整隐藏状态中提取任务相关知识。每个提示定义了答案空间中的一条独特轨迹,轨迹选择对任务表现及后续导航至关重要。我们分析了寻找最优提示的复杂性,并刻画了特定任务下提示空间的规模。理论揭示了高效提示设计的原则,表明“逐步思考”等简单自引导提示可能严重限制性能。实验表明,最优提示搜索可在推理任务上带来超过50%的性能提升,为提示工程提供了理论基础。

原文摘要 · Abstract (English)

Despite the remarkable successes of large language models (LLMs), the underlying Transformer architecture has inherent limitations in handling complex reasoning tasks. Chain-of-thought (CoT) prompting has emerged as a practical workaround, but most CoT-based methods rely on a single, generic prompt such as "think step by step", with no task-specific adaptation. These approaches expect the model to discover an effective reasoning path on its own, forcing it to search through a vast prompt space. In contrast, several studies have explored task-specific prompt designs to boost performance. However, these designs are typically developed through trial and error, lacking theoretical grounding. As a result, prompt engineering remains largely ad hoc and unguided. In this paper, we provide a theoretical framework that explains why some prompts succeed while others fail. We show that prompts function as selectors, extracting task-relevant information from the model's full hidden state during CoT reasoning. Each prompt defines a unique trajectory through the answer space, and the choice of trajectory is crucial for task performance and future navigation within the space. We analyze the complexity of finding optimal prompts and characterize the size of the prompt space for a given task. Our theory reveals principles behind effective prompt design and shows that naive CoT-using self-guided prompts like "think step by step"-can severely hinder performance. Through experiments, we show that optimal prompt search can lead to more than a 50% improvement on reasoning tasks, providing a theoretical foundation for prompt engineering.

提示工程推理优化理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。