用强化学习动态调整训练中关键令牌权重，让大模型自我迭代进化。
ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution
- 在预训练中段引入强化学习，动态重加权关键语义令牌。
- 数学、代码等10项基准平均提升3%，后继训练保持超2%增益。
- 无需教师模型，适合追求持续优化的大模型研发团队。
大型语言模型的标准训练流程通常是单向的，从预训练到后训练。然而，后训练阶段的洞察是否能反向提升预训练基础模型的能力仍待探索。本文提出一种双向反馈机制：通过强化学习（RL）微调的模型增强基础模型，进而提升后续后训练表现，且无需专门训练的教师模型。我们分析了训练动态，发现训练中段（退火期）是模型能力的关键转折点，该阶段使用高质量语料库并采用快速衰减的学习率。基于此，提出ReMiT（强化学习引导的中段训练），利用强化学习模型的推理先验，在中段训练中动态重加权令牌，优先关注对推理至关重要的内容。实验表明，ReMiT在10个预训练基准上平均提升3%，并在整个后训练流程中维持超过2%的性能增益。结果验证了可迭代、自强化的模型演化路径。
原文摘要 · Abstract (English)
Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process--where insights from post-training retroactively improve the pre-trained foundation--remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher or reference model. To realize this, we analyze training dynamics and identify the mid-training (annealing) phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training). Specifically, ReMiT leverages the reasoning priors of RL-tuned models to dynamically reweight tokens during the mid-training phase, prioritizing those pivotal for reasoning. Empirically, ReMiT achieves an average improvement of 3\% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2\% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling continuous and self-reinforcing evolution of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。