让模型通过类比推理,用更聪明的检索提升解题能力
Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning

- 用推理价值而非语义相似性来检索类比题目
- 在AIME数学竞赛上,准确率提升7.1到2.8个百分点
- 适合需要复杂逻辑推理的任务和想提升模型泛化能力的研究者
检索增强生成(RAG)已成为将语言模型与外部知识结合的标准方法,但传统的基于词法或语义相似性的检索不适用于复杂推理任务:语义相近的问题可能需要完全不同解法,而表面不同问题可能共享相同推理模式。本文提出检索增强强化微调(RA-RFT),一种后训练框架,教会语言模型通过类比进行推理。RA-RFT使用黄金相关性蒸馏训练一个检索器,按预期推理收益排序上下文而非语义重叠;随后通过强化微调方法,用检索到的类比示范对策略模型进行微调,使模型学会利用可验证结果奖励的推理轨迹。我们进一步分析检索内容多样性,发现推理感知检索能提供互补的解题策略,为每个问题提供不同的推理支架。在多个挑战性数学推理基准测试中,RA-RFT持续优于标准强化微调方法。例如,在AIME 2025上,对Qwen3-1.7B和Qwen3-4B分别提升平均@32准确率7.1和2.8点,表明推理感知检索是与奖励设计或训练课程改进并行的补充提升路径。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has become a standard mechanism for grounding language models in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches language models to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. We further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies that provide distinct reasoning scaffolds for individual problems. Across challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively -- suggesting that reasoning-aware retrieval is a complementary axis of improvement and orthogonal to advances in reward design or training curricula.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。