揭示RL让大模型泛化的核心机制:保留基础表征,动态演化关键特征。
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models

- 通过特征空间对齐,对比分析强化学习与监督微调的特征演化差异。
- 发现强化学习维持更少、更持续演化的特征,且存在一组跨任务通用的紧凑特征集。
- 这些特征具因果作用:增强可提升性能,关闭则导致泛化能力下降,适合模型优化研究者。
基于强化学习(RL)的后训练常能提升大语言模型(LLM)在训练域外的推理能力,而监督微调(SFT)则易导致泛化能力遗忘。然而,两者机制差异尚不明确。为此,我们提出一种特征级机制分析方法,在控制实验中从同一基座模型出发,对齐内部激活并分析后训练过程中的特征演化。结果表明,SFT早期快速引入大量高度特化的特征并迅速稳定,而RL则引发更克制、持续演化的特征变化,基本保留基座模型的表征。针对RL成功但基座模型失败的样本,我们识别出一组紧凑、任务无关的特征,直接中介跨任务泛化。特征干预实验证实其因果性:禁用这些特征显著降低RL模型泛化性能,放大则提升基座模型表现。代码已开源。
原文摘要 · Abstract (English)
Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models' representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models' generalization performance, while amplifying them improves base models' performance. The code is available at https://github.com/danshi777/RL-generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。