解决自对弈训练中虚假多样性问题，提升大模型推理能力持续进化
R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
- 引入记忆增强惩罚和技能感知度量，识别并抑制重复模式
- 在10个推理任务上持续优于现有方法，避免早期优势衰减
- 适合研究自对弈强化学习与大模型推理能力提升的学者
自对弈通过挑战者-求解者迭代循环提升大模型推理能力：挑战者生成针对求解者能力的问题，求解者基于生成数据优化以扩展推理技能。然而，现有框架如R-Zero常出现非持续性提升，早期进步随自对弈进行而退化。我们识别出关键失败模式——多样性幻觉：训练信号看似多样，实则收敛于重复底层模式。表现为（1）局部多样性幻觉：仅在批次内强制多样性，导致跨迭代模式循环；（2）表面多样性幻觉：问题表层变化但需相似推理技能。为此提出R-Diverse，包含两项协同创新：记忆增强惩罚（MAP），使用持久记忆库防止跨迭代重复；技能感知度量（SAM），依据所动用的推理技能而非问题表面变化评估多样性。在10个数学与通用推理基准上，R-Diverse维持更长时间增益，且持续优于先前自对弈方法。代码已开源：https://github.com/Gengsheng-Li/R-Diverse。
原文摘要 · Abstract (English)
Self-play bootstraps LLM reasoning through an iterative Challenger-Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, Diversity Illusion, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) Local Diversity Illusion, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) Surface Diversity Illusion, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: Memory-Augmented Penalty (MAP), which uses a persistent memory bank to discourage recycling across iterations, and Skill-Aware Measurement (SAM), which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods. Code is available at https://github.com/Gengsheng-Li/R-Diverse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。