arXiv:2608.30319cs.CLcs.AI2026-08中稿 · EMNLP

通过隐藏层方向引导,实现专业化大模型推理时的安全对齐。

Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering

论文配图:Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
图 1 · 摘自论文原文
  • 用跨家族模型提取安全方向,操控基础模型隐藏状态
  • 在不破坏领域能力的前提下,安全性能提升最高达22.2%
  • 适合需安全对齐但无法重训的专业化LLM场景

针对专业化微调的大语言模型,推理时对齐可有效提升因专业化导致的安全性下降问题,且无需大量计算资源,是易用、即插即用的补充方案。然而现有方法在提升安全性时往往破坏领域能力。我们发现根本原因是专用基础模型与通用引导模型具备正交专长,导致引导信号不可靠,主要表现为停止标记干扰:引导模型倾向于继续生成,覆盖基础模型的正确停止决策。为此,我们提出CREST,一种推理时对齐方法,通过从任意家族的引导模型中提取安全方向,直接操纵基础模型的隐藏表示,完全避开基于词元的结构限制。CREST在弱化安全性的专业化模型上提升安全性,同时保留领域能力及已对齐模型的安全性,在安全基准测试中优于基线高达22.2%。代码已开源:https://github.com/DecayingSeart/CREST。

原文摘要 · Abstract (English)

Large language models (LLMs) finetuned for specialized domains represent crucial high-impact applications. Inference-time alignment improves safety degraded from specialization finetuning without requiring substantial computational resources, complementing finetuning-based methods with an easy-to-use, plug-and-play solution. However, existing inference-time methods fail to reliably improve safety without disrupting domain capability. We identify the root cause as complementary expertise orthogonality: specialized base models and general-domain guidance models have orthogonal competencies, making the guidance signal unreliable for specialized generation. This primarily manifests as stop token interference, where the guidance model's tendency toward continuation overrides the base model's decision to stop, burying correct answers under guidance-induced continuation. To address this problem, we propose CREST, an inference-time alignment method that steers base model hidden representations using safety directions extracted from a guidance model of any family, avoiding token-level structural limitations entirely. CREST improves safety where specialization has weakened it while preserving both domain-specific capability and the safety of already well-aligned models, outperforming baselines by up to 22.2\% on safety benchmarks. Our code is available at: https://github.com/DecayingSeart/CREST.

大模型对齐推理优化安全生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。