通过调整模型中间层向量方向，控制大模型拒绝回答行为。
Scaling laws for activation steering with Llama 2 models and refusal mechanisms
- 用对比样本提取向量方向，在前中层注入以改变输出。
- 模型越大，控制效果越弱，70B模型效果最差。
- 负面引导比正面引导更有效，适用于安全对齐研究。
随着大型语言模型复杂度提升，许多非主流对齐技术的有效性变得不确定。本文基于激活转向与对比激活添加（CAA）方法，利用Llama 2系列模型（7B、13B、70B）探究了模型规模对CAA效果的影响。CAA通过对比样本（如‘仇恨’与‘爱’）在残差流向量空间中识别理想方向，并在前向传播中将其注入残差流，直接操纵模型输出。实验聚焦于拒绝行为的问答任务，发现：1）CAA在早期至中期层注入效果最佳；2）模型规模越大，效果越弱；3）负向引导在所有模型尺寸下均产生更显著影响。
原文摘要 · Abstract (English)
As large language models (LLMs) evolve in complexity and capability, the efficacy of less widely deployed alignment techniques are uncertain. Building on previous work on activation steering and contrastive activation addition (CAA), this paper explores the effectiveness of CAA with model scale using the family of Llama 2 models (7B, 13B, and 70B). CAA works by finding desirable 'directions' in the model's residual stream vector space using contrastive pairs (for example, hate to love) and adding this direction to the residual stream during the forward pass. It directly manipulates the residual stream and aims to extract features from language models to better control their outputs. Using answer matching questions centered around the refusal behavior, we found that 1) CAA is most effective when applied at early-mid layers. 2) The effectiveness of CAA diminishes with model size. 3) Negative steering has more pronounced effects than positive steering across all model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。