arXiv:2608.08224cs.LG2026-08

提出新方法提升大模型多任务推理能力,突破训练瓶颈。

Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training

论文配图:Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training
图 1 · 摘自论文原文
  • 用控制系数分析模型内部决策机制,区分激活与实际控制
  • 发现共享激活不等于共享控制,揭示控制坍缩问题
  • 设计正则化方法实现控制解耦,提升数学、代码等多任务表现

强化学习后训练使大语言模型具备复杂推理能力。但基准得分仅反映性能提升,无法揭示内部变化或能力在任务间的分配情况。现有解释认为成功源于更强更丰富的电路激活,我们挑战此观点,提出将激活与控制分离:激活的电路未必主导奖励增益。借鉴代谢控制分析,定义后训练控制系数,构建控制矩阵与激活强度矩阵。发现跨任务控制集中是共享控制瓶颈,激活与控制集中差异为激活-控制差距。高共享激活可与任务特异性控制共存;小差距表明控制坍缩至共享方向,丧失任务特异性。为此,我们引入共享控制瓶颈正则项,提出控制多样性强化微调(CD-RFT)。由于精确正则项需二阶自动微分,与Flash Attention不兼容,我们推导出一阶近似,最坏情况下开销低于8%。在Qwen2.5-7B上,CD-RFT实现最大控制解耦,多任务能力优于匹配的GRPO,涵盖数学、代码与逻辑任务。无KL项变体在pass@1上领先,含KL惩罚变体在大k pass@k覆盖率上更优,而传统方法会削弱该指标。结果表明共享控制瓶颈既是机制诊断工具,也是有效训练正则,且控制解耦与能力提升可迁移至Llama-3.2-3B。

原文摘要 · Abstract (English)

Reinforcement learning post-training unlocks complex reasoning in LLMs. Yet benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks. A representative interpretability line attributes the success of RL fine-tuning to stronger and more diverse circuit activation. We challenge this activation-centered account by separating activation from control: an activated circuit need not control the post-training reward gain. Adapting Metabolic Control Analysis, we define the Post-training Control Coefficient to measure component control over reward gain and arrange these coefficients by task family into a control matrix, paired with an activation-magnitude matrix. We call cross-task control concentration the Shared Control Bottleneck and the difference between activation and control concentration the Activation-Control Gap. This reveals that highly shared activations can coexist with task-specific control, while a small gap indicates that control has collapsed onto a shared direction and lost task specificity. To reduce this collapse, we regularize the post-training loss with the Shared Control Bottleneck and propose Control-Diverse Reinforcement Fine-Tuning (CD-RFT). The exact regularizer gradient requires second-order automatic differentiation incompatible with flash attention, so we derive a first-order proxy with worst-case overhead below eight percent. On Qwen2.5-7B, CD-RFT achieves the largest control decoupling and improves multi-task capability over matched GRPO across mathematics, code, and logic. The no-KL variant leads on pass@1, and the KL-penalized variant leads on large-k pass@k coverage that KL otherwise degrades. Together, these results show that the Shared Control Bottleneck is both a mechanistic diagnostic and a training regularizer, and that control decoupling and capability gains transfer to Llama-3.2-3B.

强化学习大模型推理控制分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。