arXiv:2605.19577cs.CL2026-05被引 3

开源长文本强化学习数据集,提升模型长上下文理解能力

GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment

论文配图:GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
图 1 · 摘自论文原文
  • 基于长文本能力分类构建2.3万条开源样本,覆盖9类任务
  • 在相同训练设置下超越闭源数据集,性能媲美顶尖大模型
  • 提出新奖励优化方法,有效解决多任务奖励差异问题

我们提出GoLongRL,一个完全开源、以能力为导向的长上下文强化学习后训练方案,支持可验证奖励(RLVR)。现有长上下文强化学习方法通常依赖复杂检索路径构建数据,导致任务覆盖单一、奖励设计无法反映实际需求。本工作贡献两点:(1) 能力导向的数据构建与全开源发布。我们公开发布包含23,000条RLVR样本的数据集、完整构建流程及全部训练代码。依据长上下文能力分类体系,数据涵盖9种任务类型,每类匹配自然评估指标。样本由知名语料库中的精选开源内容与基于真实文档(如书籍、论文、多轮对话)生成的合成问答对构成。在相同基础GRPO设置下,仅使用该数据集即超越闭源QwenLong-L1.5数据集。此外,基于该数据训练的Qwen3-30B-A3B模型,在长上下文表现上达到DeepSeek-R1-0528和Qwen3-235B-A22B-Thinking-2507的水平,表明更广的任务覆盖与更多样化的奖励机制显著促进长上下文能力提升。(2) TMN-Reweight实现异构多任务优化。为应对异构奖励带来的优化挑战,我们提出TMN-Reweight,结合任务级均值归一化以对齐跨任务奖励尺度,以及难度自适应加权以获得更可靠的优势估计。该方法在保持各项通用能力的基础上,进一步提升了平均性能。

原文摘要 · Abstract (English)

We present GoLongRL, a fully open-source, capability-oriented post-training recipe for long-context reinforcement learning with verifiable rewards (RLVR). Existing long-context RL methods often treat data construction as a matter of designing increasingly complex retrieval paths, leading to homogeneous task coverage and reward formulations that inadequately reflect practical long-context requirements. Our work offers two contributions. (1) Capability-oriented data construction with full open release. We openly release a dataset of 23K RLVR samples, the complete construction pipeline, and all training code. Guided by a taxonomy of long-context capabilities, the dataset spans 9 task types, each paired with its natural evaluation metric. It comprises curated open-source samples from established corpora and synthetic samples whose QA pairs are generated from real source documents such as books, academic papers, and multi-turn dialogues. Under the same vanilla GRPO setup, our dataset alone outperforms the closed-source QwenLong-L1.5 dataset. Moreover, our Qwen3-30B-A3B model trained on this data delivers long-context performance comparable to DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507, suggesting that broader coverage and greater reward diversity substantially benefit long-context capability improvement. (2) TMN-Reweight for heterogeneous multitask optimization. To address optimization challenges from heterogeneous rewards, we propose TMN-Reweight, which combines task-level mean normalization for cross-task reward scale alignment with difficulty-adaptive weighting for more reliable advantage estimation. TMN-Reweight further improves average performance over vanilla GRPO, with general capabilities preserved or improved across reported evaluations.

强化学习长文本开源数据多任务优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。