统一解释语言模型控制方法,揭示控制强度与生成质量的权衡
Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics
- 将多种控制方法统一为受控信号引发的动态权重更新
- 发现控制强度越强,目标偏好越高但生成质量越低
- 提出新方法SPLIT,在保持质量前提下提升控制效果
控制大型语言模型的方法,如局部权重微调、基于LoRA的适配和基于激活的干预,常被孤立研究,难以建立联系。本文提出一种统一视角,将这些干预视为由控制信号引发的动态权重更新,并构建统一的偏好-效用分析框架。该框架将控制效果分解为偏好(向目标概念的倾向性)和效用(生成内容的一致性和任务有效性),并使用极性配对对比样本在共享对数似然尺度上进行测量。在多种方法中均观察到偏好与效用之间的稳定权衡:更强的控制提升偏好,但可预测地降低效用。通过激活流形视角解释这一现象:控制沿目标概念方向移动表征以增强偏好,而效用下降主要源于干预使表征偏离模型有效生成流形。最后,基于此分析提出新控制方法SPLIT,可在保持更高效用的同时提升偏好。代码已公开。
原文摘要 · Abstract (English)
Methods for controlling large language models (LLMs), including local weight fine-tuning, LoRA-based adaptation, and activation-based interventions, are often studied in isolation, obscuring their connections and making comparison difficult. In this work, we present a unified view that frames these interventions as dynamic weight updates induced by a control signal, placing them within a single conceptual framework. Building on this view, we propose a unified preference-utility analysis that separates control effects into preference, defined as the tendency toward a target concept, and utility, defined as coherent and task-valid generation, and measures both on a shared log-odds scale using polarity-paired contrastive examples. Across methods, we observe a consistent trade-off between preference and utility: stronger control increases preference while predictably reducing utility. We further explain this behavior through an activation manifold perspective, in which control shifts representations along target-concept directions to enhance preference, while utility declines primarily when interventions push representations off the model's valid-generation manifold. Finally, we introduce a new steering approach SPLIT guided by this analysis that improves preference while better preserving utility. Code is available at https://github.com/zjunlp/EasyEdit/blob/main/examples/SPLIT.md.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。