发现大模型强化学习中训练与推理不一致会引发训练崩溃
Diagnosing Training Inference Mismatch in LLM Reinforcement Learning
- 构建零偏差诊断环境，分离出训练-推理差异问题
- 微小的词元级数值差异即可导致训练完全崩溃
- 揭示该问题本质是系统级扰动，需优先考虑
现代大模型强化学习系统将生成回溯与策略优化分开进行。这两个阶段本应基于相同模型权重产生完全一致的词元概率，但实现上的差异可能导致相同序列在不同阶段分配不同概率值，从而引发训练-推理不一致（TIM）。TIM难以检测，因其与非策略漂移和常见稳定机制交织在一起。本文提出一个零偏差诊断设置（VeXact），首次隔离出TIM现象，并证明微小的词元级数值差异可独立导致训练崩溃。进一步分析表明，TIM改变了有效优化问题，我们识别出一组可缓解该问题的修复方案。结果表明，TIM并非无害的数值噪声，而是一种应作为一阶因素考虑的系统级扰动，对大模型强化学习的稳定性具有关键影响。
原文摘要 · Abstract (English)
Modern LLM RL systems separate rollout generation from policy optimization. These two stages are expected to produce token probabilities that match exactly. However, implementation differences can make them assign different values to the same sequence under the same model weights, inducing Training-Inference Mismatch (TIM). TIM is difficult to inspect because it is entangled with off-policy drift and common stabilization mechanisms. In this work, we isolate TIM in a zero-mismatch diagnostic setting (VeXact), and show that small token-level numerical disagreements can independently cause training collapse. We further show that TIM changes the effective optimization problem, and identify a set of remedies that could mitigate TIM. Our results suggest that TIM is not benign numerical noise, but a systems-level perturbation that should be treated as a first-order factor in analyzing LLM RL stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。