提出无需调参的鲁棒提示干预方法,提升大模型行为控制效果。
Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions

- 联合训练因子与方向,避免推理时手动调参
- 仅干预提示词片段的提示专属向量优于全序列干预
- 在可控性与生成质量间取得更好平衡,适合实际部署
近期,转向向量(SVs)已成为一种高效轻量的大语言模型行为调控方法,其中微调后的SV比无优化的更有效。然而,现有微调方法存在两大局限:一是需对每个向量单独选择转向因子以平衡调控效果与生成质量;二是采用全序列转向向量(FSSV),即使因子选择得当,仍因过度干预导致生成质量下降。为此,我们提出联合训练因子与方向,使推理阶段无需后验调参。基于神经网络缩放理论,发现中等偏大的初始化规模与学习率有助于联合训练的稳定与高效。针对第二个问题,借鉴表示微调思想,引入仅作用于部分提示词的提示专属向量(PrOSV)。实证结果表明,在AxBench上,结合本联合训练方案的PrOSV优于传统FSSV。此外,PrOSV在通用模型能力与对抗鲁棒性之间表现更优。
原文摘要 · Abstract (English)
Recently, steering vectors (SVs) have emerged as an effective and lightweight approach to steer behaviors of large language models (LLMs), among which fine-tuned SVs are more effective than optimization-free ones. However, current approaches to fine-tuned SVs suffer from two limitations. First, they require careful selection of steering factors on a per-SV basis to balance steering effectiveness and generation quality at inference time. Second, they operate as full-sequence SVs (FSSVs), which can sacrifice generation quality regardless of factor selection due to excessive intervention on the model generation process. To address the first limitation, we propose joint training of steering factors and directions, such that post-hoc factor selection is no longer required. Using neural network scaling theory, we find that moderately large initialization sizes and learning rates for steering factors are essential for stability and efficiency of joint training. To tackle the second limitation, we draw inspiration from representation fine-tuning and introduce Prompt-only SV (PrOSV), an SV that intervenes only on a few prompt tokens. Our empirical results show that PrOSV outperforms traditional FSSVs on AxBench when using our joint training scheme. We also find that PrOSV achieves a better tradeoff between general model utility and adversarial robustness than FSSV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。