发现一种特征可让大模型在不重训练下精准控制工具使用行为
Localizing RL-Induced Tool Use to a Single Crosscoder Feature
- 通过交叉编码器筛选出与强化学习相关的关键特征
- 使模型工具调用准确率提升31.1个百分点，且能迁移到基础模型
- 适合想实现零成本行为控制的AI系统开发者
通过强化学习微调，语言模型内部表征被重塑以支持自主行为（如工具调用），但其机制仍不清晰。本文发现，专用特征交叉编码器（DFC）能从Qwen2.5-3B中提取一组紧凑的、由强化学习引入的特征，这些特征直接决定工具调用能力。在48组交叉编码器超参数测试中，编码-解码重构使强化学习模型的工具调用正确率提升31.1±9.7个百分点，并将工具调用能力被动转移至冻结的基础模型，提升6.8±5.0个百分点，这种现象称为‘能力溢出’。研究结果表明，通过DFC分块可将强化学习引入的能力集中到一个最小且可调控的特征集，实现对智能大模型运行时行为的精准控制。
原文摘要 · Abstract (English)
Fine-tuning through RL reshapes the internal representations of language models to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that $\textit{Dedicated Feature Crosscoders (DFC)}$ isolate a compact set of RL-specific features that mediate tool-calling capability in $\texttt{Qwen2.5-3B}$. Across a $48$-crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model's tool correctness by $+31.1 \pm {9.7}$ pp and passively transfers tool-calling ability to the frozen base model by $+6.8 \pm 5.0$ pp which we call a $\textit{capability spillover}$. Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。