提出几何约束方法,让大模型同时执行多个语义控制而不崩溃。
GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs
- 通过几何约束实现多语义方向无损叠加。
- 三重语义注入下GSM8K准确率保持98%,远超未约束的4%。
- 无需训练,适配3B到31B多种模型架构。
激活引导在不重新训练的情况下,通过修改推理时的中间隐藏状态来控制模型行为。现有方法仅支持单向注入;当多个语义方向叠加且无约束时,模型会崩溃。我们发现该崩溃由两个独立作用的来源构成:分布偏差,即加性扰动在层间累积导致激活超出训练分布;方向干扰,即非正交语义向量叠加时相互抑制。这两者定义了任何免训练多方向干预必须满足的设计约束。作为这一原则的具体实现,我们提出GEMS,一种免训练方法,将每个来源映射为相应几何约束:保持范数的加权叠加和定向注意力路径注入应对分布偏差,实时正交化处理方向干扰。在GSM8K上,同时注入三个非数学方向,准确率仍维持在98%(基线92%),而无约束叠加降至4%;在Wikitext-2上,相同注入仅增加2.2%的PPL。组件消融实验验证了各约束的因果作用,层级探针显示正交化信号能穿越前馈网络并保有语义特异性。定性控制效果可跨3B至31B模型架构迁移。
原文摘要 · Abstract (English)
Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining. Existing methods handle only single-direction injection; when multiple semantic directions are superposed without constraints, the model collapses. We show that this collapse decomposes into two independently acting sources: distributional deviation, where additive perturbations accumulate in norm across layers and drive activations outside the training distribution, and directional interference, where non-orthogonal semantic vectors mutually dampen when superposed. These two sources define the design constraints that any training-free multi-directional intervention must address. As one instantiation of these principles, we propose GEMS, a training-free method that maps each source to a corresponding geometric constraint: norm-preserving weighted superposition and targeted attention-pathway injection for distributional deviation, and real-time orthogonalization for directional interference. On GSM8K, injecting three concurrent non-mathematical directions preserves accuracy at 98% (baseline 92%), while unconstrained addition collapses to 4%; on Wikitext-2, the same injection incurs only 2.2% PPL increase. Component ablation isolates the causal role of each constraint, and layer-level probes confirm that orthogonalized signals survive the FFN pathway and reach the output distribution with semantic specificity. Qualitative steering effects transfer across architectures from 3B to 31B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。