无需人工标注,通过自对弈强化学习提升长文本推理能力
SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models
- 用提问者、回答者、验证者三角色自循环训练模型
- 在6个基准上平均提升7.6分,优于有标注数据微调的模型
- 适合想提升长文本推理的模型开发者和研究者
大语言模型在长文本推理方面进展缓慢,主要因处理长文本难度高,且缺乏可靠的人工标注与可程序验证的奖励信号。本文提出SPELL框架,一种多角色自对弈强化学习方法,实现无标签、可扩展的长文本推理优化。该框架在同一模型中集成提问者、响应者和验证者三种角色,形成持续自我改进的闭环:提问者从原始文档生成问题并提供参考答案;响应者基于文档作答;验证者评估响应内容与参考答案的语义等价性,生成奖励信号以指导训练。为稳定训练,引入自动化课程机制,逐步增加文档长度,并动态调整问题难度以匹配模型能力。在六个长文本基准上的实验表明,SPELL能持续提升多种大模型性能,优于同等规模经大规模标注数据微调的模型。尤其在强推理模型Qwen3-30B-A3B-Thinking上,其pass@8指标平均提升7.6分,显著提高性能上限,展现出向更强大模型扩展的潜力。代码已开源。
原文摘要 · Abstract (English)
Progress in long-context reasoning for large language models (LLMs) has lagged behind other recent advances. This gap arises not only from the intrinsic difficulty of processing long texts, but also from the scarcity of reliable human annotations and programmatically verifiable reward signals. In this paper, we propose SPELL, a multi-role self-play reinforcement learning framework that enables scalable, label-free optimization for long-context reasoning. SPELL integrates three cyclical roles-questioner, responder, and verifier-within a single model to enable continual self-improvement. The questioner generates questions from raw documents paired with reference answers; the responder learns to solve these questions based on the documents; and the verifier evaluates semantic equivalence between the responder's output and the questioner's reference answer, producing reward signals to guide continual training. To stabilize training, we introduce an automated curriculum that gradually increases document length and a reward function that adapts question difficulty to the model's evolving capabilities. Extensive experiments on six long-context benchmarks show that SPELL consistently improves performance across diverse LLMs and outperforms equally sized models fine-tuned on large-scale annotated data. Notably, SPELL achieves an average 7.6-point gain in pass@8 on the strong reasoning model Qwen3-30B-A3B-Thinking, raising its performance ceiling and showing promise for scaling to even more capable models. Our code is available at https://github.com/Tongyi-Zhiwen/Qwen-Doc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。