arXiv:2601.07411cs.LGcs.AI2026-01

通过低秩参数编辑,精准删减大模型特定能力而不影响其他功能。

SCALPEL: Selective Capability Ablation via Low-rank Parameter Editing for Large Language Model Interpretability Analysis

  • 将能力建模为跨层分布的低秩参数子空间,实现细粒度控制。
  • 在BLiMP任务上成功移除特定能力,同时保持通用语言能力不变。
  • 适合研究模型内部机制或需要可控能力调整的开发者使用。

大型语言模型在多个领域表现卓越,但其在医疗、法律及自主决策系统中的应用受限于对内部机制的理解不足。传统方法依赖梯度或激活分析定位关键模块,假设能力与组件一一对应,然而神经计算复杂:单个模块可能支持多种能力,单一能力也分布于多个模块。为此,本文提出SCALPEL(基于低秩参数编辑的选择性能力消融框架),将能力表示为分布于多层模块的低秩参数子空间。核心思想是:能力可通过跨层的低秩修改来表征,从而实现精准移除而不干扰其他能力。通过训练LoRA适配器,在保留通用语言建模质量的前提下降低正确与错误答案的区分度,SCALPEL识别出负责特定能力的低秩表示,并确保与其他能力解耦。在BLiMP涵盖的多样能力与语言任务上实验表明,该方法能有效移除目标能力并保留通用能力,揭示能力在参数空间中具有低秩结构,且可通过针对性参数干预实现选择性消融,为理解大模型能力编码提供细粒度视角。

原文摘要 · Abstract (English)

Large language models excel across diverse domains, yet their deployment in healthcare, legal systems, and autonomous decision-making remains limited by incomplete understanding of their internal mechanisms. As these models integrate into high-stakes systems, understanding how they encode capabilities has become fundamental to interpretability research. Traditional approaches identify important modules through gradient attribution or activation analysis, assuming specific capabilities map to specific components. However, this oversimplifies neural computation: modules may contribute to multiple capabilities simultaneously, while single capabilities may distribute across multiple modules. These coarse-grained analyses fail to capture fine-grained, distributed capability encoding. We present SCALPEL (Selective Capability Ablation via Low-rank Parameter Editing for Large language models), a framework representing capabilities as low-rank parameter subspaces rather than discrete modules. Our key insight is that capabilities can be characterized by low-rank modifications distributed across layers and modules, enabling precise capability removal without affecting others. By training LoRA adapters to reduce distinguishing correct from incorrect answers while preserving general language modeling quality, SCALPEL identifies low-rank representations responsible for particular capabilities while remaining disentangled from others. Experiments across diverse capability and linguistic tasks from BLiMP demonstrate that SCALPEL successfully removes target capabilities while preserving general capabilities, providing fine-grained insights into capability distribution across parameter space. Results reveal that capabilities exhibit low-rank structure and can be selectively ablated through targeted parameter-space interventions, offering nuanced understanding of capability encoding in LLMs.

大模型解释低秩编辑能力解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。