用外部知识库+推理优化，让小模型也能精准解决专业问题
RARE: Retrieval-Augmented Reasoning Modeling
- 把专业知识存外面，训练时只学推理模式
- 80亿参数小模型比GPT-4高20%准确率
- 适合需要精准推理的医疗、法律等专业场景
领域智能需要专业知识和复杂推理能力，但大语言模型在参数受限下易产生知识幻觉且推理不足。受教育学中的布卢姆分类法启发，我们提出检索增强推理建模（RARE），将知识存储与推理优化分离：将领域知识外化为可检索源，在训练中内化特定领域的推理模式。通过在训练提示中注入检索到的知识并使用掩码损失，使学习目标从机械记忆转变为情境化推理，避免了对大量参数的依赖，专注于发展高阶认知能力。大量实验表明，轻量级的RARE训练模型（如Llama-3.1-8B）可达到顶尖性能，准确率超越检索增强的GPT-4和DeepSeek-R1约20%。RARE建立了一种范式变革：可维护的外部知识库与紧凑的推理优化模型协同工作，共同推动更可持续的领域智能发展。
原文摘要 · Abstract (English)
Domain-specific intelligence demands specialized knowledge and sophisticated reasoning for problem-solving, posing significant challenges for large language models (LLMs) that struggle with knowledge hallucination and inadequate reasoning capabilities under constrained parameter budgets. Inspired by Bloom's Taxonomy in educational theory, we propose Retrieval-Augmented Reasoning Modeling (RARE), a novel paradigm that decouples knowledge storage from reasoning optimization. RARE externalizes domain knowledge to retrievable sources and internalizes domain-specific reasoning patterns during training. Specifically, by injecting retrieved knowledge into training prompts with masked losses, RARE transforms learning objectives from rote memorization to contextualized reasoning. It enables models to bypass parameter-intensive memorization and prioritize the development of higher-order cognitive processes. Extensive experiments demonstrate that lightweight RARE-trained models (e.g., Llama-3.1-8B) could achieve state-of-the-art performance, surpassing retrieval-augmented GPT-4 and DeepSeek-R1 up to approximately 20\% accuracy. RARE establishes a paradigm shift where maintainable external knowledge bases synergize with compact, reasoning-optimized models, collectively driving more scalable domain-specific intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。