用奖励信号训练稀疏自编码器,发现分离效果主要来自完成度而非推理质量。
Reward-Informed Sparse Autoencoders and the Solution-Completeness Confound

- 用高/低奖励轨迹划分数据,训练稀疏自编码器提取特征
- 仅16,384个特征中的少数能区分好坏完成,但主要反映完成形式
- 提示:奖励信号易被完成度误导,适合研究解释性方法的局限性
稀疏自编码器(SAEs)可将语言模型激活分解为稀疏且可解释的特征。本文通过强化学习产生的奖励信号来引导其训练:将GRPO轨迹分为高奖励(好)和低奖励(坏)的推理延续,对这些激活训练标准的JumpReLU SAE,再分析其好坏分离能力。在Llama-3.1-8B上,16,384个特征中的一个稀疏子集实现了良好分离(选中特征的轮廓系数0.79,全代码仅为0.005)。但对照实验表明,这种分离主要源于解题完成度,而非推理质量——一个TF-IDF分类器即可实现AUC 0.75–0.83,三个结构线索(长度、封闭推理块、框出答案)单独已达AUC 0.70(99%的好完成与69%的坏完成有框出答案)。未见过奖励的通用SAE完全无法分离类别(轮廓系数0.01,无判别特征),说明0.79是针对该标注信号的过拟合结果,而非奖励无关字典所恢复的结构性模式。因此,本文同时呈现方法与对照组:奖励过滤是一种低成本、无需标签的重用强化学习信号方式,但其揭示的多为完成形式;仅两个可读特征(符号数学、程序化与评价性语言)仍具意义,但仅为示例而非独立推理证据。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, and an appealing way to aim them at reasoning is to curate their data with a signal reinforcement learning already produces: the reward. We build such a reward-informed SAE (RI-SAE): we split GRPO trajectories into high-reward ("good") and low-reward ("bad") reasoning continuations, train a standard JumpReLU SAE on their activations, and then ask what the resulting good/bad separation actually measures. On Llama-3.1-8B a sparse subset of the 16,384 features does separate the classes (silhouette 0.79 on the selected features versus 0.005 for the full code), but a control battery shows the separation is largely solution completeness rather than reasoning quality: a TF-IDF text classifier already splits the classes (AUC 0.75--0.83), and three structural cues alone (length, a closed reasoning block, and a boxed answer) reach AUC 0.70 (99% of good versus 69% of bad completions are boxed). A generic SAE that never saw the reward does not separate the classes at all (silhouette 0.01, no discriminative features), so the 0.79 is in-sample fitting of this curated signal rather than structure that a reward-blind dictionary recovers. We therefore present the recipe and its control battery together: reward filtering is a cheap, label-free way to reuse RL signals for interpretability, but most of what it surfaces is completion form. Two discriminative features are still readable (symbolic mathematics; procedural and evaluative language), which we take as illustrative rather than as isolated reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。