让大模型自己玩猜谜，通过角色切换提升复杂问题解决能力。
AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
- 模型轮流扮演拆解问题和求解答案的角色，协同训练。
- 在10个数据集上平均准确率提升7.6%，小模型性能超大模型。
- 无需中间标注，适合需要高效推理的金融等专业场景。
搜索增强型大语言模型在复杂推理任务中表现不佳，主要受限于多跳检索效率低和推理能力不足。本文提出AceSearcher，一种合作式自对弈框架，让单一大模型交替扮演分解者（拆解复杂问题）和求解者（整合检索信息生成答案）两个角色。该框架结合多样任务的监督微调与以最终答案准确率为目标的强化微调，无需中间标注。在三个高推理强度任务、共10个数据集上的实验表明，AceSearcher显著优于现有基线，平均精确匹配率提升7.6%。尤其在文档级金融推理任务中，AceSearcher-32B仅用不到5%的参数量，达到DeepSeek-V3的性能；即使在1.5B和8B的小规模下，其表现也超过参数量高达9倍的现有搜索增强模型，展现出卓越的效率与有效性。代码将发布于https://github.com/ritaranx/AceSearcher和https://huggingface.co/AceSearcher。
原文摘要 · Abstract (English)
Search-augmented LLMs often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. We propose AceSearcher, a cooperative self-play framework that trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceSearcher couples supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9x more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks. Our code will be published at https://github.com/ritaranx/AceSearcher and https://huggingface.co/AceSearcher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。