只训练一个Transformer层,效果竟可媲美全参数强化学习。
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

- 通过分层实验发现,仅训练中间层即可捕获大部分强化学习收益。
- 在7个模型、3种算法上,单层训练性能接近甚至超过全参数训练。
- 中间层贡献显著,且跨任务、模型和算法的层重要性排序高度一致。
强化学习(RL)已成为大语言模型后训练的核心组件,但其在Transformer各层间的分布机制尚不明确。现有方法通常对所有参数进行均匀更新,隐含假设各层贡献相当。本文通过系统性的分层研究挑战这一假设。结果表明:仅训练单一Transformer层即可恢复全参数强化学习的绝大部分收益,甚至在某些情况下超越全参数训练。为此,我们提出“层贡献”指标,衡量单独训练某层所恢复的总改进比例。在涵盖两个模型家族(Qwen3、Qwen2.5)、三种强化学习算法(GRPO、GiGPO、Dr. GRPO)及数学推理、代码生成、智能体决策等多个任务领域的实验中,均观察到惊人稳定的模式:强化学习增益高度集中于少数甚至单个中间层,而靠近输入输出端的层贡献较小。该结构规律在不同数据集、任务、模型家族与算法间保持强相关性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。