arXiv:2509.22510cs.CL2025-09被引 4

提出新方法让大模型同时做到有用、安全、诚实,避免互相干扰。

We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong

  • 用共享表示+分支修正的方式,统一处理三大目标
  • 在LLaMA-2-7B上平均提升56.5%的综合表现
  • 单次推理完成多目标输出,适合实际部署

大型语言模型(LLM)的对齐能力对其可信部署至关重要。实践中常以有用性、无害性和诚实性(HHH)为目标进行对齐。现有方法在标准Transformer解码器中使用引导向量,但孤立优化各目标,易造成冲突;部分改进将表示复制到多路径,但独立变换导致响应不一致。安全强化学习与混合专家设计也未在推理时约束目标特定变换。为此,我们提出自适应多分支引导(AMBS),一种两阶段1对N Transformer框架:第一阶段计算一次共享隐藏表示;第二阶段将该表示复制到N条路径,相对共享参考进行更新,捕捉目标特异性偏差的同时限制发散。单次前向传播生成N个目标响应,可在解码时融合得到统一输出。在多个模型上,AMBS显著提升HHH性能,如在LLaMA-2-7B上平均达56.5%,且保持高效(189 Token/s,9 GPU小时)。

原文摘要 · Abstract (English)

Alignment of Large Language Models (LLMs) is the ability to satisfy desired objectives during generation, which is critical for trustworthy deployment. In practice, alignment is often operationalized through multiple objectives such as Helpfulness, Harmlessness, and Honesty (HHH). Prior works study alignment via steering vectors in standard Transformer decoders but treat objectives in isolation, where optimizing a single objective can overwrite others, leading to interference. Recent works attempt to address this limitation by extending steering to a 1-to-N Transformer setting by replicating representations into objective-specific pathways, but apply transformations independently, resulting in inconsistent responses across objectives. Similarly, approaches such as safe RLHF and MoE-based designs study trade-offs across objectives but do not constrain objective-specific transformations within a shared representation during inference. As a result, even aligned State-of-the-Art (SOTA) LLMs can struggle to jointly satisfy HHH objectives in complex settings. To address this, we propose Adaptive Multi-Branch Steering (AMBS), a two-stage framework in a 1-to-N Transformer setting that parameterizes objective-specific transformations relative to a shared representation. In Stage I, a shared hidden representation is computed once. In Stage II, this representation is replicated into N pathways and updated relative to a shared reference, capturing objective-specific deviations while restricting divergence. This produces N objective-specific responses within a single forward pass, which can be combined at decoding to obtain a single response across objectives. Across multiple backbones, AMBS improves performance across HHH, with consistent gains in WR, TI, and SS (e.g., Avg 56.5% on LLaMA-2-7B) while maintaining efficiency (e.g., 189 Tok/s, 9 GPU-hrs).

大模型对齐多目标优化推理效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。