用分阶段训练让大模型自我进化更稳定，不依赖人工标注也能逼近监督模型效果。
DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution
- 分两阶段训练：先生成难度可控的问题，再用外部文档辅助生成高质量伪标签
- 在9个推理基准上平均提升10.9分，三种模型均表现超越基线
- 适合追求模型自进化且无标注数据的科研与工程团队
自博弈框架在实现大语言模型自我进化方面展现出巨大潜力，但现有方法常因（i）由求解器依赖的奖励反馈导致目标非平稳，以及（ii）使用自生成伪标签引发的自举误差而出现优化不稳定。为此，我们提出DARC（解耦式非对称推理课程），一种两阶段框架以稳定自演化过程。首先，训练提问者根据显式难度等级和外部语料库生成难度校准的问题；其次，通过非对称自蒸馏机制训练求解器，其中基于文档的教师模型生成高质量伪标签，监督无法访问文档的学生求解器。实验表明，DARC具有模型无关性，在九个推理基准上平均提升10.9分，覆盖三种主干模型。其性能持续优于所有基线，并在不依赖人工标注的前提下接近全监督模型水平。代码已开源：https://github.com/RUCBM/DARC。
原文摘要 · Abstract (English)
Self-play with large language models has emerged as a promising paradigm for achieving self-improving artificial intelligence. However, existing self-play frameworks often suffer from optimization instability, due to (i) non-stationary objectives induced by solver-dependent reward feedback for the Questioner, and (ii) bootstrapping errors from self-generated pseudo-labels used to supervise the Solver. To mitigate these challenges, we introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process. First, we train the Questioner to synthesize difficulty-calibrated questions, conditioned on explicit difficulty levels and external corpora. Second, we train the Solver with an asymmetric self-distillation mechanism, where a document-augmented teacher generates high-quality pseudo-labels to supervise the student Solver that lacks document access. Empirical results demonstrate that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models. Moreover, DARC consistently outperforms all baselines and approaches the performance of fully supervised models without relying on human annotations. The code is available at https://github.com/RUCBM/DARC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。