通过实例级程序生成提升大模型推理能力，减少错误结果。
Once Upon an Input: Reasoning via Per-Instance Program Synthesis
- 为每个输入实例动态生成并优化程序，利用结构反馈改进
- 在30个基准上最高提升8.6%准确率，算法任务错误率降65.1%
- 无需任务特定引导，适配复杂推理场景如数学与视觉问答
大语言模型在零样本推理中表现优异，但在复杂多步推理任务中仍存在困难。现有方法如思维链（CoT）和程序思维（PoT）虽有改进，但在算法类任务中常产生错误解。本文提出实例级程序合成（PIPS），通过结构化反馈在实例层面生成并优化程序，无需任务特定指导或显式测试用例。同时引入置信度机制，按实例动态选择直接推理或程序合成。在三个前沿大模型及30个基准（包括Big Bench Extra Hard全部任务、视觉问答、关系推理与数学推理）上的实验表明，PIPS相较PoT和CoT分别提升绝对调和平均准确率最多8.6%和9.4%，且在算法任务上将错误程序生成减少65.1%（以Gemini-2.0-Flash为例）。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at zero-shot inference but continue to struggle with complex, multi-step reasoning. Recent methods that augment LLMs with intermediate reasoning steps such as Chain of Thought (CoT) and Program of Thought (PoT) improve performance but often produce undesirable solutions, especially in algorithmic domains. We introduce Per-Instance Program Synthesis (PIPS), a method that generates and refines programs at the instance-level using structural feedback without relying on task-specific guidance or explicit test cases. To further improve performance, PIPS incorporates a confidence metric that dynamically chooses between direct inference and program synthesis on a per-instance basis. Experiments across three frontier LLMs and 30 benchmarks including all tasks of Big Bench Extra Hard (BBEH), visual question answering tasks, relational reasoning tasks, and mathematical reasoning tasks show that PIPS improves the absolute harmonic mean accuracy by up to 8.6% and 9.4% compared to PoT and CoT respectively, and reduces undesirable program generations by 65.1% on the algorithmic tasks compared to PoT with Gemini-2.0-Flash.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。