让大模型学会推理，用离线训练替代实时搜索。
Offline Learning and Forgetting for Reasoning with Large Language Models
- 用成功与失败推理路径训练模型，实现离线推理。
- 在24点和计数谜题上成功率提升23%，推理快180倍。
- 适合追求高效推理的AI系统开发者。
利用大语言模型推理时的搜索能力可有效提升其解决复杂数学与推理问题的能力，但显著增加计算成本和推理时间，因需生成并评估多个候选解以找到可行推理路径。为此，我们提出一种方法，通过在多样化搜索方法生成的未配对成功（学习）与失败（遗忘）推理路径上微调模型，将搜索能力直接嵌入模型。我们发现，直接微调会损害模型搜索能力；通过使用较小学习率可缓解此问题。在具有挑战性的24点与计数谜题任务上，用搜索生成数据替代思维链生成数据进行离线微调，使成功率比基于推理时搜索的基线提高约23%，同时推理时间减少180倍。此外，我们的学习与遗忘目标在性能上持续优于监督微调与基于偏好方法。
原文摘要 · Abstract (English)
Leveraging inference-time search in large language models has proven effective in further enhancing a trained model's capability to solve complex mathematical and reasoning problems. However, this approach significantly increases computational costs and inference time, as the model must generate and evaluate multiple candidate solutions to identify a viable reasoning path. To address this, we propose an effective approach that integrates search capabilities directly into the model by fine-tuning it on unpaired successful (learning) and failed reasoning paths (forgetting) derived from diverse search methods. A key challenge we identify is that naive fine-tuning can degrade the model's search capability; we show this can be mitigated with a smaller learning rate. Extensive experiments on the challenging Game-of-24 and Countdown arithmetic puzzles show that, replacing CoT-generated data with search-generated data for offline fine-tuning improves success rates by around 23% over inference-time search baselines, while reducing inference time by 180$\times$. On top of this, our learning and forgetting objective consistently outperforms both supervised fine-tuning and preference-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。