用强化学习提升大模型长文本推理能力,效果媲美更大模型。
LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts
- 通过插入隐藏问题的UUID链生成高难度长文本任务
- 训练后模型在128K长度任务上准确率提升23.5%以上
- 适合需要强长程推理的AI研究者和开发者
长上下文推理对大语言模型至关重要。尽管强化学习(RL)能通过激发“顿悟”提升短上下文推理,但长上下文所需的高级思维模式仍待探索,且高难度RL数据稀缺。本文提出LoongRL,一种面向长上下文高级推理的数据驱动强化学习方法。核心是KeyChain:将短多跳问答转化为高难度长上下文任务,通过插入包含大量干扰文档的UUID链,使真实问题被隐藏。解题需模型逐步追踪链条、识别真问题、检索相关事实并推理作答。在KeyChain数据上进行RL训练,涌现出“规划-检索-推理-验证”的新思维模式,泛化能力远超训练长度。16K长度训练的模型可有效解决128K任务,无需高昂的全长度RL回放成本。在Qwen2.5-7B和14B上,长上下文多跳问答准确率分别提升+23.5%和+21.1%。最终的LoongRL-14B得分达74.2,接近o3-mini(74.5)和DeepSeek-R1(74.9)。同时提升长上下文检索性能,通过全部128K针堆测试,并保持短上下文推理能力。
原文摘要 · Abstract (English)
Reasoning over long contexts is essential for large language models. While reinforcement learning (RL) enhances short-context reasoning by inducing "Aha" moments in chain-of-thought, the advanced thinking patterns required for long-context reasoning remain largely unexplored, and high-difficulty RL data are scarce. In this paper, we introduce LoongRL, a data-driven RL method for advanced long-context reasoning. Central to LoongRL is KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among large collections of distracting documents. Solving these tasks requires the model to trace the correct chain step-by-step, identify the true question, retrieve relevant facts and reason over them to answer correctly. RL training on KeyChain data induces an emergent plan-retrieve-reason-recheck reasoning pattern that generalizes far beyond training length. Models trained at 16K effectively solve 128K tasks without prohibitive full-length RL rollout costs. On Qwen2.5-7B and 14B, LoongRL substantially improves long-context multi-hop QA accuracy by +23.5% and +21.1% absolute gains. The resulting LoongRL-14B reaches a score of 74.2, rivaling much larger frontier models such as o3-mini (74.5) and DeepSeek-R1 (74.9). It also improves long-context retrieval, passes all 128K needle-in-a-haystack stress tests, and preserves short-context reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。