用强化学习让大模型更擅长长文本推理,效果媲美顶级模型。
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
- 通过渐进式上下文扩展+课程引导强化学习,稳定训练过程。
- 在7个长文档问答任务上超越OpenAI-o3-mini和Qwen3-235B,接近Claude-3.7-Sonnet-Thinking。
- 适合需要处理长篇文档、复杂推理的场景,如法律、科研分析。
近期的大规模推理模型(LRMs)通过强化学习(RL)展现出强大的推理能力,但主要集中在短上下文任务。将这些模型扩展到长上下文推理仍面临重大挑战。为此,我们首次形式化了长上下文推理的强化学习范式,并识别出训练效率低下与优化不稳定的痛点。提出QwenLong-L1框架,通过渐进式上下文扩展,将短上下文模型适配至长上下文场景:先进行预热监督微调建立初始策略,再采用课程引导的分阶段强化学习稳定策略演化,并引入难度感知回溯采样策略激励探索。在七个长上下文文档问答基准测试中,QwenLong-L1-32B的表现超越OpenAI-o3-mini和Qwen3-235B-A22B,性能接近Claude-3.7-Sonnet-Thinking,成为当前最先进的长上下文推理模型之一。本工作推动了可应用于信息密集环境的鲁棒长上下文推理模型的发展。
原文摘要 · Abstract (English)
Recent large reasoning models (LRMs) have demonstrated strong reasoning capabilities through reinforcement learning (RL). These improvements have primarily been observed within the short-context reasoning tasks. In contrast, extending LRMs to effectively process and reason on long-context inputs via RL remains a critical unsolved challenge. To bridge this gap, we first formalize the paradigm of long-context reasoning RL, and identify key challenges in suboptimal training efficiency and unstable optimization process. To address these issues, we propose QwenLong-L1, a framework that adapts short-context LRMs to long-context scenarios via progressive context scaling. Specifically, we utilize a warm-up supervised fine-tuning (SFT) stage to establish a robust initial policy, followed by a curriculum-guided phased RL technique to stabilize the policy evolution, and enhanced with a difficulty-aware retrospective sampling strategy to incentivize the policy exploration. Experiments on seven long-context document question-answering benchmarks demonstrate that QwenLong-L1-32B outperforms flagship LRMs like OpenAI-o3-mini and Qwen3-235B-A22B, achieving performance on par with Claude-3.7-Sonnet-Thinking, demonstrating leading performance among state-of-the-art LRMs. This work advances the development of practical long-context LRMs capable of robust reasoning across information-intensive environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。