通过词汇随机遮蔽，打破大模型自演化中的问题固化，提升推理能力。
Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
- 用随机遮蔽提议者输出词元，防止其陷入固定表达模式。
- 在数学推理任务中使提议多样性持续提升，解题器平均得分提高4.4分。
- 适合研究自演化训练、提示工程或大模型能力提升的开发者。
自演化式自我对弈（co-evolutionary self-play）中，一个语言模型生成题目，另一个求解，有望实现无需人工干预的课程学习。但实践中早期即出现瓶颈：提议者逐渐收敛到满足奖励函数的狭窄问题分布，导致课程内容单一，无法有效训练求解者，使循环停滞。本文提出词汇遮蔽（vocabulary dropout），一种轻量级干预策略，在策略训练和课程生成阶段随机掩码提议者的输出词元概率。该掩码为硬性且非静态，使提议者无法锁定特定词元序列。在 Qwen3-4B 与 Qwen3-8B 上基于 R-Zero 进行数学推理训练，词汇遮蔽在整个训练过程中维持了提议者在词汇、语义和功能层面的多样性，并使求解器平均提升 +4.4 分，尤其在竞赛级基准上表现显著。显式动作空间约束可发挥类似经典自对弈中游戏规则的作用，维持语言模型自演化过程的生产性。词汇遮蔽是实现此类约束的一种简洁方式。
原文摘要 · Abstract (English)
Co-evolutionary self-play, where one language model generates problems and another solves them, promises curriculum learning without human supervision. The promise breaks down early in practice. The proposer converges to a narrow distribution of problems that satisfy the reward function, and the collapsed curriculum teaches the solver little, stalling the loop. We introduce vocabulary dropout, a lightweight intervention that randomly masks the proposer's output logits during both policy training and curriculum generation. The mask is hard and non-stationary, so the proposer cannot lock into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, vocabulary dropout sustains proposer diversity throughout training across lexical, semantic, and functional measures, and improves the solver by an average of +4.4 points at 8B with the largest gains on competition-level benchmarks. Explicit action-space constraints, filling the structural role that game rules fill in classical self-play, can keep co-evolution in language productive. Vocabulary dropout is one simple way to impose them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。