用强化学习统一检索与推理，让大模型更聪明地查资料、做判断。
UR$^2$: Unify RAG and Reasoning through Reinforcement Learning
- 通过难度感知课程动态决定何时调用检索，避免无效查询。
- 结合固定知识库与实时生成摘要，提升信息获取效率与准确性。
- 适用于医疗、数学等复杂领域，适合想提升模型推理能力的研究者。
大型语言模型在两种互补范式上表现突出：检索增强生成（RAG）用于知识锚定，基于可验证奖励的强化学习（RLVR）用于复杂推理。然而现有融合方法多局限于开放域问答且检索设置固定，难以泛化到更广泛领域。为此，我们提出UR$^2$（统一检索与推理），一个通用强化学习框架，可动态协调检索与推理。UR$^2$引入两项关键设计：难度感知课程，仅对困难样本触发检索；混合知识访问策略，结合领域专用离线语料与实时生成的LLM摘要。两者协同缓解检索与推理间的失衡，增强对噪声信息的鲁棒性。在开放域问答、MMLU-Pro、医学与数学推理任务上的实验表明，基于Qwen-2.5-3/7B和LLaMA-3.1-8B的UR$^2$持续优于现有RAG与强化学习基线，在多个基准上达到GPT-4o-mini和GPT-4.1-mini水平。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG) for knowledge grounding and Reinforcement Learning from Verifiable Rewards (RLVR) for complex reasoning. However, existing attempts to unify these paradigms remain narrow in scope, typically limited to open-domain QA with fixed retrieval settings, which constrains generalization to broader domains. To address this limitation, we propose UR$^2$ (Unified RAG and Reasoning)), a general reinforcement learning framework that dynamically coordinates retrieval and reasoning. UR$^2$ introduces two key designs: a difficulty-aware curriculum that selectively invokes retrieval only for challenging instances, and a hybrid knowledge access strategy that combines domain-specific offline corpora with on-the-fly LLM-generated summaries. Together, these components mitigate the imbalance between retrieval and reasoning and improve robustness to noisy information. Experiments on open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks show that UR$^2$, built on Qwen-2.5-3/7B and LLaMA-3.1-8B, consistently outperforms existing RAG and RL baselines, and achieves performance comparable to GPT-4o-mini and GPT-4.1-mini on several benchmarks. Our code is available at https://github.com/Tsinghua-dhy/UR2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。