无需外部标注，通过双策略自演化提升语言模型在开放任务上的表现。
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
- 设计双策略框架：挑战者生成文档任务，求解器多轮检索作答。
- 在8个基准上性能提升最高达+10.4分，媲美需9000条人工提示的训练。
- 适用于开放任务与短问答，且可自动生成评分标准，降低人工依赖。
自洽博弈可无监督训练语言模型。但现有方法依赖可验证答案，使开放任务仍需人工提示或前沿模型评判。我们提出SCOPE，一种无需数据的自洽博弈框架，协同进化两个策略：挑战者生成基于文档的开放任务，求解者通过多轮检索作答。使用初始模型的冻结副本作为自评判者，从源文档生成任务特定评分标准，并据此评估求解者回答。在三个7-8B指令微调模型（Qwen2.5、Qwen3、OLMo-3）上，SCOPE在八个基准上性能提升最高达+10.4分，达到或超过基于约9000条人工提示训练的GRPO_data表现。尽管仅在开放任务上训练，其在七个保留的短问答基准上也实现最高+13.8分提升，在三模型上均超越GRPO_data。消融实验表明，协同进化挑战者对维持任务难度在求解者能力边界至关重要；性能提升来自检索与合成双重优化，贡献比例依任务而异；评分标准生成质量是自评判的瓶颈。
原文摘要 · Abstract (English)
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。