用稀疏自编码器特征动态控制大模型，实现每一步的可解释干预。
Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features
- 通过强化学习训练策略，选择激活特定特征来逐令牌控制输出。
- 在多个基准上提升性能，同时生成每步干预日志以追踪决策路径。
- 适合研究模型内部机制、可解释性与安全可控性的研究人员。
稀疏自编码器（SAEs）能将语言模型激活分解为可解释特征，但现有方法仅能识别哪些特征被激活，无法判断哪些特征在放大后会改变模型输出。本文提出控制强化学习（CRL），通过训练策略在每个词元处选择SAE特征进行引导，生成可解释的干预日志：学习到的策略能识别出放大后影响输出的特征。自适应特征掩码促进多样特征发现，同时保持单特征可解释性。该框架带来新分析能力：分支点追踪定位决定输出正确性的关键词元；批评轨迹分析分离策略局限与价值估计误差；层间比较揭示早期层存在句法特征，后期层呈现语义特征。在Gemma 2 2B模型上，于MMLU、BBQ、GSM8K、HarmBench和XSTest多个数据集均实现性能提升，并提供逐词元干预日志。结果表明，学习到的特征引导是一种机制可解释性工具，为静态特征分析补充了动态干预探针。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) decompose language model activations into interpretable features, but existing methods reveal only which features activate, not which change model outputs when amplified. We introduce Control Reinforcement Learning (CRL), which trains a policy to select SAE features for steering at each token, producing interpretable intervention logs: the learned policy identifies features that change model outputs when amplified. Adaptive Feature Masking encourages diverse feature discovery while preserving singlefeature interpretability. The framework yields new analysis capabilities: branch point tracking locates tokens where feature choice determines output correctness; critic trajectory analysis separates policy limitations from value estimation errors; layer-wise comparison reveals syntactic features in early layers and semantic features in later layers. On Gemma 2 2B across MMLU, BBQ, GSM8K, HarmBench, and XSTest, CRL achieves improvements while providing per-token intervention logs. These results establish learned feature steering as a mechanistic interpretability tool that complements static feature analysis with dynamic intervention probes
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。