对比发现，强化学习比监督微调更少破坏模型内部计算回路。
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
- 提出头级电路脆弱性度量，量化微调对模型内部结构的破坏程度。
- 监督微调适应更快但导致87%以上原始回路被破坏，强化学习则保留更多基础回路。
- 适合关注大模型稳定性与机制可解释性的研究者参考。
微调大型语言模型常引发对先前能力的灾难性遗忘。近期研究表明，强化学习（RL）比监督微调（SFT）更能保留原有能力，归因于策略梯度更新更贴近基线策略。本文将此行为解释延伸至机制层面，探究RL优势是否体现在对内部计算回路的更好保护。我们引入差分电路脆弱性（differential circuit vulnerability），在头级别衡量回路在微调下的退化程度，并在适配科学问答任务的Qwen2.5-3B-Instruct上比较了RL与SFT。结果揭示明显的机制权衡：SFT虽更快适应新任务，但造成显著的回路破坏和先验能力丢失；而RL虽适应较慢，却能保留更大比例的基础回路。这些发现表明，回路保留可能是RL更具鲁棒性以应对灾难性遗忘的原因。代码已开源：https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。