通过可擦除强化学习修复多跳推理错误，提升大模型搜索增强能力
Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
- 识别并擦除错误推理步骤，原位重生成逻辑链
- 3B和7B模型在多个数据集上指标提升超8%和11%
- 适合需要可靠多步推理的复杂问答场景
尽管搜索增强的大语言模型表现出色，但在复杂多跳推理中的可靠性仍有限。主要源于三个根本挑战：分解错误、检索遗漏和推理错误，任一环节失败都会导致最终答案出错。本文提出可擦除强化学习（ERL）框架，将脆弱的推理过程变为鲁棒流程。ERL 明确识别错误步骤，将其擦除并原位重生成，阻止缺陷逻辑在推理链中传播。经 ERL 训练的模型（ESearch）在 HotpotQA、MuSiQue、2Wiki、Bamboogle 上实现显著提升：3B 模型达到 +8.48% EM 和 +11.56% F1，7B 模型达 +5.38% EM 和 +7.22% F1，优于先前 SOTA。结果表明，可擦除强化学习为大模型的鲁棒多步推理提供了范式革新。
原文摘要 · Abstract (English)
While search-augmented large language models (LLMs) exhibit impressive capabilities, their reliability in complex multi-hop reasoning remains limited. This limitation arises from three fundamental challenges: decomposition errors, where tasks are incorrectly broken down; retrieval missing, where key evidence fails to be retrieved; and reasoning errors, where flawed logic propagates through the reasoning chain. A single failure in any of these stages can derail the final answer. We propose Erasable Reinforcement Learning (ERL), a novel framework that transforms fragile reasoning into a robust process. ERL explicitly identifies faulty steps, erases them, and regenerates reasoning in place, preventing defective logic from propagating through the reasoning chain. This targeted correction mechanism turns brittle reasoning into a more resilient process. Models trained with ERL, termed ESearch, achieve substantial improvements on HotpotQA, MuSiQue, 2Wiki, and Bamboogle, with the 3B model achieving +8.48% EM and +11.56% F1, and the 7B model achieving +5.38% EM and +7.22% F1 over previous state-of-the-art(SOTA) results. These findings suggest that erasable reinforcement learning provides a powerful paradigm shift for robust multi-step reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。