用线性控制理论实现大模型推理时的精准行为调控。
Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control
- 将大模型推理建模为时变线性系统，利用层间雅可比矩阵设计反馈控制器。
- 在不训练的前提下实现毒性、真实性等属性的精细调节，性能优于现有方法。
- 适用于多种模型规模和任务，提供可证明的调控误差边界，适合安全可控生成场景。
推理阶段的大模型对齐方法，特别是激活值调控，提供了一种无需微调即可直接修改生成过程激活值的替代方案。然而，现有方法通常依赖非前瞻性的干预，忽略了扰动在变压器层间的传播特性，且缺乏在线误差反馈，导致控制效果不佳，属于开环控制。为此，我们通过实验证明，尽管变压器模块具有非线性结构，但多个大模型架构与尺度下的层间动态仍可被局部线性模型良好近似。基于此性质，我们将大模型推理建模为线性时变动力系统，并引入经典的线性二次调节器（LQR），利用层间雅可比矩阵计算闭环反馈控制器，以最小计算开销实现激活值向目标语义点的精准引导，无需离线训练。我们还推导出目标跟踪误差的理论界，实现了对调控性能的形式化保证。结合一种新型自适应语义特征目标信号，该方法在不同模型、规模和任务上均表现出鲁棒且细粒度的行为控制能力，包括在毒性、真实性、拒绝响应及任意概念上的最先进调控表现，显著超越基线调控方法。代码已开源：https://github.com/trustworthyrobotics/lqr-activation-steering
原文摘要 · Abstract (English)
Inference-time LLM alignment methods, particularly activation steering, offer an alternative to fine-tuning by directly modifying activations during generation. Existing methods, however, often rely on non-anticipative interventions that ignore how perturbations propagate through transformer layers and lack online error feedback, resulting in suboptimal, open-loop control. To address this, we show empirically that, despite the nonlinear structure of transformer blocks, layer-wise dynamics across multiple LLM architectures and scales are well-approximated by locally-linear models. Exploiting this property, we model LLM inference as a linear time-varying dynamical system and adapt the classical linear quadratic regulator to compute feedback controllers using layer-wise Jacobians, steering activations toward desired semantic setpoints in closed-loop with minimal computational overhead and no offline training. We also derive theoretical bounds on setpoint tracking error, enabling formal guarantees on steering performance. Using a novel adaptive semantic feature setpoint signal, our method yields robust, fine-grained behavior control across models, scales, and tasks, including state-of-the-art modulation of toxicity, truthfulness, refusal, and arbitrary concepts, surpassing baseline steering methods. Our code is available at: https://github.com/trustworthyrobotics/lqr-activation-steering
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。