通过关键路径识别,精准定位并修改少量核心特征,提升大模型知识修正效果。
Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steering

- 基于因果依赖识别关键特征,构建高效引导路径
- 仅修改少量特征即提升准确率18%,优于传统批量方法
- 适合需要高精度、低干扰模型编辑的场景
基于稀疏自编码器(SAE)的引导方法常用于解决大语言模型中的知识冲突问题,但现有方法多采用大规模特征修改,依赖相关性分析识别特征,因相关性不准及忽略特征间交互,导致引入大量冗余特征,增加噪声并削弱引导效果。实证研究发现,仅对少数识别出的特征进行引导即可达到甚至超越原有性能。为此,我们提出关键路径识别(KPI)方法,通过识别与上游和下游特征具有强因果依赖关系的关键特征,构建关键路径,实现更少的特征修改。该方法将SAE引导从数量驱动转向质量导向,提升了模型编辑的精确性与可解释性。在存在知识冲突的RAG任务中,实验表明本方法相比最优批量引导基线平均提升准确率18%,有效过滤冗余特征,缓解副作用,验证了关键路径在引导中的核心作用。
原文摘要 · Abstract (English)
Sparse autoencoder (SAE)-based steering has been widely used to address knowledge conflicts by guiding LLMs to be more faithful to the contextual knowledge. Existing methods usually perform mass steering, which modifies a large batch of SAE features identified via correlation-based methods. However, due to the inaccurate correlation and the neglected feature interactions, mass steering methods fail to precisely identify the features that play the key roles in steering and introduce a large number of redundant ones, which add noise and weaken the steering effects. Our empirical studies reveal that steering only a small subset of the identified features can achieve comparable or even better performance. Motivated by this finding, we propose Key Path Identification (KPI), a novel method that identifies key steering features characterized by strong causal dependencies with both upstream and downstream features. From these features, KPI constructs key paths and steers through less feature modifications. In this way, KPI advances SAE-based steering from quantity-driven to quality-focused, offering a perspective for more precise and interpretable model editing. Experiments in RAG tasks with knowledge conflicts show that our method improves the accuracy by 18% on average compared to the best baseline of mass steering, effectively filtering redundant features, alleviating side effects and demonstrating the core role of key paths in steering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。