用演员表演类比大模型，显著提升解谜能力
LLMs as Method Actors: A Model for Prompt Engineering and Architecture
- 将大模型视为演员，提示词为剧本，优化推理流程
- 在连接词谜题中，正确率从41%提升至86%
- 适合需要精细推理链的复杂任务设计
我们提出“方法演员”这一心理模型，用于指导大模型的提示工程与架构设计。在此模型中，大模型被视为演员，提示词为剧本与提示，模型输出则为表演。我们将该模型应用于改进GPT-4o在《纽约时报》连接词谜题（Connections）上的表现，该任务被先前研究认为是评估大模型推理能力的难点。实验表明，采用“方法演员”策略后，解决率达86%，显著优于基础提示（27%）和思维链（41%）。我们还测试了OpenAI新推出的复杂推理模型o1-preview：一次性求解正确率为79%，分步多次调用可实现100%正确。引入“方法演员”提示架构后，o1-preview完美解题率从76%提升至87%。
原文摘要 · Abstract (English)
We introduce "Method Actors" as a mental model for guiding LLM prompt engineering and prompt architecture. Under this mental model, LLMs should be thought of as actors; prompts as scripts and cues; and LLM responses as performances. We apply this mental model to the task of improving LLM performance at playing Connections, a New York Times word puzzle game that prior research identified as a challenging benchmark for evaluating LLM reasoning. Our experiments with GPT-4o show that a "Method Actors" approach can significantly improve LLM performance over both a vanilla and "Chain of Thoughts" approach. A vanilla approach solves 27% of Connections puzzles in our dataset and a "Chain of Thoughts" approach solves 41% of puzzles, whereas our strongest "Method Actor" approach solves 86% of puzzles. We also test OpenAI's newest model designed specifically for complex reasoning tasks, o1-preview. When asked to solve a puzzle all at once, o1-preview solves 79% of Connections puzzles in our dataset, and when allowed to build puzzle solutions one guess at a time over multiple API calls, o1-preview solves 100% of the puzzles. Incorporating a "Method Actor" prompt architecture increases the percentage of puzzles that o1-preview solves perfectly from 76% to 87%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。