无需人工干预,自动实现大模型行为调控的轻量级后训练方法
Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Models
- 全自动构建激活向量,无需提示词设计或特征标注
- 在18个任务中显著提升行为类任务性能,最高提升34.8%
- 可无缝集成现有方法,适合快速部署与大规模应用
语言模型通常通过权重调整或提示工程进行后训练以获得期望能力,但前者耗时昂贵,后者控制不精准且需反复试错。激活调控(AS)虽具成本低、速度快、可控性强的优势,但现有方法依赖手工构造提示对或繁琐的特征标注,使用不便。本文提出无痛激活调控(PAS),一种完全自动化的方法,仅需标注数据集即可实现激活调控,无需提示构造、特征标注或人工参与。我们在三个开源模型(Llama3.1-8B-Instruct、DeepSeek-R1-Distill-8B、Nous-Hermes-2)和18个任务上评估,发现PAS能稳定提升行为类任务表现,但对智力导向任务无效。内省变体(iPAS)效果最强,分别在偏见(+10.1%)、道德性(+5.2%)和对齐度(+34.8%)上提升显著。PAS还能在上下文学习(ICL)和监督微调(SFT)基础上进一步增益。该方法生成快速、轻量的激活向量,可低成本训练、便捷存储并按需启用。结果揭示了激活调控适用边界及实用部署路径。
原文摘要 · Abstract (English)
Language models (LMs) are typically post-trained for desired capabilities and behaviors via weight-based or prompt-based steering, but the former is time-consuming and expensive, and the latter is not precisely controllable and often requires manual trial-and-error. While activation steering (AS) promises a cheap, fast, and controllable alternative to the two existing post-training methods, current AS techniques require hand-crafted prompt pairs or labor-intensive feature annotation, making them more inconvenient than the plug-and-play methods such as Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT). We introduce Painless Activation Steering (PAS), a family of fully automated methods that make AS readily usable with any given labeled dataset, with no need for prompt construction, feature labeling, or human intervention. We evaluate PAS on three open-weight models (Llama3.1-8B-Instruct, DeepSeek-R1-Distill-8B, and Nous-Hermes-2) and 18 tasks; we find that PAS reliably improves performance for behavior tasks, but not for intelligence-oriented tasks. The introspective variant (iPAS) delivers the strongest causal steering effects (10.1% on Bias, 5.2% on Morality, and 34.8% on Alignment). We also show PAS delivers additional gains on top of In-Context Learning (ICL) and SFT. PAS constructs a fast, lightweight activation vector that can be cheaply trained, easily stored, and activated at will. Our results provide a characterization of where AS helps, where it fails, and how to deploy it as a practical, automated LM post-training option.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。