发现并修复大模型奖励模型的偏好不稳定性问题
Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders
- 用稀疏自编码器识别导致偏好的脆弱特征
- 在多个基准上降低错误偏好率，且不重训练模型
- 适合关注模型可靠性和对齐安全的研究者
大型语言模型中的偏好学习依赖奖励模型模拟人类判断，但这些模型常因微小语义不变的输入变化产生矛盾偏好。本文在三种语义保持扰动（改写、模式注入、后门触发）下分析表示层的不稳定性，发现其源于对预测性但脆弱特征的过度依赖，称为不稳定特征。通过稀疏自编码器（SAE）在稀疏隐空间中将正常与扰动输入激活的模式区分开来。基于此可分性，提出两种方法：SAE特征引导，在推理时抑制异常激活特征；SAE残差修正，学习自适应调整以恢复正确偏好。实验显示，该方法显著减少有害性和幻觉基准上的错误偏好，同时保持原有任务性能和通用能力，无需重训练原奖励模型。代码与数据已开源。
原文摘要 · Abstract (English)
Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit preference instability, producing contradictory preference assignments in response to subtle, meaning-preserving input variations. We analyze this instability at the representation level under three semantic-preserving perturbation types: paraphrasing, pattern injection, and backdoor triggers. We attribute this instability to over-reliance on predictive yet brittle features, which we term unstable features, and isolate them via Sparse Autoencoders (SAEs) in a sparse latent space where benign and perturbed inputs activate distinctly separable patterns. Building on this separability, we propose two SAE-based instability mitigation strategies: SAE Feature Steering, which identifies and suppresses anomalously activated features at inference, and SAE Residual Correction, which learns adaptive adjustments over SAE features to restore correct preferences. Our methods substantially reduce incorrect preference assignments on harmlessness and hallucination benchmarks while preserving benign performance and general utility on other tasks, without retraining the reward model. Our code and data are available in \url{https://github.com/shunchang-liu/pisa}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。