让大模型用外部工具改进思维过程，提升推理准确性。
Teaching LLMs to Refine with Tools
- 引入外部工具辅助修正大模型的思维链推理过程。
- 通过两阶段训练使模型在反馈中有效优化结果，显著提升准确率。
- 适合需要高精度推理的场景，如数学、代码生成等任务。
大型语言模型（LLMs）可通过反馈自我改进，实现迭代训练或测试时的精炼。然而，现有方法大多局限于同一推理格式内的修正，可能导致无效甚至错误的修正行为。本文提出CaP，一种利用外部工具对同模型或不同模型生成的思维链（CoT）响应进行精炼的新方法。CaP采用两阶段训练：监督微调后接基于DPO变体的偏好优化。实验表明，偏好优化在实现有效精炼中起关键作用。此外，我们对比了多种采样策略，以在推理时高效结合思维链与工具。结果证明，CaP在跨推理模式精炼和高效推理方面具有潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。