用强化学习训练可复用的提示策略，让黑箱大模型更聪明地完成多步推理和工具使用。
Prompting Policies for Multi-step Reasoning and Tool-Use in Black-box LLMs with Iterative Distillation of Experience
- 通过迭代经验蒸馏，让轻量提示模型优化大模型的提示。
- 在逻辑推理任务上从55%提升至90%，工具使用任务从74%升至91%。
- 适合研究高效提示工程与自动化推理的开发者与研究人员。
随着与冻结的“黑箱”大语言模型交互成为主流，提示工程从经验性尝试转变为关键优化挑战。我们提出一种基于强化学习的框架，通过迭代经验蒸馏训练可学习的提示策略。在此架构中，一个轻量级提示模型被优化以最大化对更大、冻结的工作者模型的任务特定奖励。通过利用耦合标量奖励与密集文本批评的对比经验缓冲区，该方法将迭代提示优化有效转化为单次策略权重。实验分析聚焦于 Big Bench Extra Hard (BBEH) 与 Tau-bench 套件，涵盖多种多步推理与工具使用任务。结果显示显著提升：逻辑密集型推理任务性能从55%提高到90%，工具使用任务从74%提升至91%。我们还分析了提示结构的演化过程，展示了策略如何发现专用算法启发式。与最先进的进化基线（如 GEPA）相比，迭代蒸馏在更高样本效率下实现了更优性能。
原文摘要 · Abstract (English)
The shift toward interacting with frozen, "black-box" Large Language Models (LLMs) has transformed prompt engineering from a heuristic exercise into a critical optimization challenge. We propose a Reinforcement Learning (RL) framework for training learned prompting policies via iterative distillation of experience. In this architecture, a lightweight prompter model is optimized to maximize task-specific rewards for a larger, frozen worker LLM. By utilizing a contrastive experience buffer that couples scalar rewards with dense textual critiques, our approach effectively amortizes iterative prompt refinement into single-shot policy weights. Our experimental analysis focuses on the Big Bench Extra Hard (BBEH) and Tau-bench suites, covering a diverse range of multi-step reasoning and tool-use tasks. We demonstrate significant gains, improving performance from 55% to 90% in logic-intensive reasoning and 74% to 91% in tool-use tasks. Furthermore, we analyze the structural evolution of prompts, demonstrating how the policy discovers specialized algorithmic heuristics. We provide comprehensive comparisons against state-of-the-art evolutionary baselines like GEPA, showing that iterative distillation achieves superior performance with higher sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。