为奖励模型打造可解释性工具,揭示其内部运作机制。
reward-lens: A Mechanistic Interpretability Library for Reward Models

- 基于奖励头权重向量构建可解释性分析框架
- 发现线性归因与因果修补效果无显著相关性
- 适合研究大模型对齐与奖励机制的开发者
每个经过RLHF训练的语言模型都受到奖励模型的影响,但现有的可解释性工具(如逻辑透镜、直接逻辑归因、激活修补、稀疏自编码器)原本针对的是以词汇解码投影为基础的生成式大模型。奖励模型采用标量回归头,破坏了这些工具的适用性。本文提出reward-lens,一个开源库,将这套工具适配到奖励模型,核心思想是:奖励头的权重向量 $w_r$ 是所有可解释性问题的自然轴线。该库提供奖励透镜、组件归因、三种模式的激活修补、奖励劫持探测套件、TopK SAE特征归因、跨模型比较,以及五种理论驱动的扩展(扭曲指数、感知分歧的修补、错位级联检测、奖励项冲突分析、概念向量分析)。通过十种方法适配协议,支持Llama、Mistral、Gemma-2和ArmoRM多目标头,也兼容任意HuggingFace序列分类模型。在两个生产级奖励模型上验证,共使用约695个RewardBench样本对。核心实证发现为负面:线性归因无法预测因果修补效果(在Skywork上平均斯皮尔曼相关系数 $ρ= -0.256$,在ArmoRM上为 $-0.027$)。框架将这一不一致视为需暴露的特性,而非缺陷,强调观测与因果视角应并列且可直接比较。
原文摘要 · Abstract (English)
Every RLHF-trained language model is shaped by a reward model, yet the mechanistic interpretability toolkit -- logit lens, direct logit attribution, activation patching, sparse autoencoders -- was built for generative LLMs whose primitives all project onto a vocabulary unembedding. Reward models replace that with a scalar regression head, breaking each tool. We present reward-lens, an open-source library that ports this toolkit to reward models, organised around one observation: the reward head's weight vector $w_r$ is the natural axis for every interpretability question. The library provides a Reward Lens, component attribution, three-mode activation patching, a reward-hacking probe suite, TopK SAE feature attribution, cross-model comparison, and five theory-grounded extensions (distortion index, divergence-aware patching, misalignment cascade detection, reward-term conflict analysis, concept-vector analysis). A ten-method adapter protocol covers Llama, Mistral, Gemma-2, and ArmoRM multi-objective heads, with a generic adapter for any HuggingFace sequence classification model. We validate on two production reward models across ~695 RewardBench pairs. The central empirical finding is negative: linear attribution does not predict causal patching effects (mean Spearman $ρ= -0.256$ on Skywork, $-0.027$ on ArmoRM). The framework treats this disagreement as a property to expose, not a bug -- motivating a design that keeps observational and causal views first-class and directly comparable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。