用轻量控制器动态调节大模型生成内容,精准阻止有害输出。
Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs
- 设计轻量控制器实时分析模型中间激活,预测全局缩放与分层权重。
- 在 ToxicChat 等测试中,拒绝率显著提升,且不修改原模型参数。
- 适用于需安全可控生成的场景,如对话系统、内容审核等。
控制大语言模型(LLM)产生不当内容的行为,传统方法依赖昂贵的微调。激活调控提供了一种推理时的替代方案,但现有方法缺乏细粒度和自适应能力。本文提出一种轻量可训练的控制器网络,在推理阶段集成使用。该控制器观察模型特定中间激活,预测全局缩放因子与层间权重,动态调节基于预计算“拒绝方向”向量的调控补丁强度,应用于各层生成过程。控制器在有害与良性提示的激活上训练,学习差异化干预策略,仅对有害输入激活调控。在 ToxicChat 及 In-The-Wild Jailbreak Prompts 安全基准测试中,本方法显著提高拒绝率,实现无参数修改的定向行为调整。实验基于 Llama-3.1-8B、Llama-3.2-1B 与 Mistral-7B 模型,结果优于现有方法,证明了推理时高效、细粒度控制的有效性。
原文摘要 · Abstract (English)
Controlling undesirable Large Language Model (LLM) behaviors, such as the generation of unsafe content or failing to adhere to safety guidelines, often relies on costly fine-tuning. Activation steering provides an alternative for inference-time control, but existing methods typically lack fine-grained, adaptive mechanisms. We introduce a novel approach using a lightweight, trainable controller network integrated during inference. This controller network observes specific intermediate LLM activations and predicts both a global scaling factor and layer-specific weights. The predicted global scaling factor and layer-specific weights then dynamically modulate the intensity of a steering patch, derived from a pre-computed "refusal direction" vector, applied across the LLM's layers during generation. Trained on activations from both harmful and benign prompts, our controller learns to discriminatively apply nuanced, layer-aware interventions, activating steering primarily for harmful inputs. Experiments using safety benchmarks like ToxicChat & In-The-Wild Jailbreak Prompts demonstrate that our weighted steering controller significantly increases refusal rates compared to the base LLM, achieving targeted behavioral modification without altering the original model parameters. Our experiments with Llama-3.1-8B, Llama-3.2-1B & Mistral-7B show our approach outperforms existing methods, presenting an efficient and adaptive method for fine-grained control over LLM behavior at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。