arXiv:2511.16324cs.CLcs.AI2025-11AAAI被引 2

无需微调即可让开源大模型更懂人意,动态调整输出分布。

SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning

  • 通过指令驱动重分配输出概率,实现推理时的无训练对齐。
  • 在8个不同模型上平均提升64.4%助人度、30%诚实度和11.5%安全性。
  • 轻量高效,支持个性化偏好控制,适合实际部署场景。

随着大语言模型的快速发展,其在真实应用中的部署日益广泛。模型需在多样任务、用户偏好与实际场景中保持稳定表现,但如何在不进行昂贵重训练或大量监督的前提下,使模型输出与人类意图对齐,仍是核心挑战。为此,本文提出 SDA(Steering-Driven Distribution Alignment),一种无需训练、模型无关的对齐框架,专为开源 LLM 设计。SDA 在推理阶段根据用户定义的对齐指令动态调整模型输出概率分布,增强模型行为与人类意图的一致性。该方法轻量、资源高效,兼容多种开源模型,可独立使用或与训练式对齐策略结合。此外,支持个性化偏好对齐,灵活控制输出行为。实验证明,SDA 在8个不同规模和来源的开源模型上,于三个关键对齐维度(3H:助人度、诚实度、无害性)均取得显著提升,平均增益分别为64.4%、30%和11.5%,验证了其有效性与泛化能力。

原文摘要 · Abstract (English)

With the rapid advancement of large language models (LLMs), their deployment in real-world applications has become increasingly widespread. LLMs are expected to deliver robust performance across diverse tasks, user preferences, and practical scenarios. However, as demands grow, ensuring that LLMs produce responses aligned with human intent remains a foundational challenge. In particular, aligning model behavior effectively and efficiently during inference, without costly retraining or extensive supervision, is both a critical requirement and a non-trivial technical endeavor. To address the challenge, we propose SDA (Steering-Driven Distribution Alignment), a training-free and model-agnostic alignment framework designed for open-source LLMs. SDA dynamically redistributes model output probabilities based on user-defined alignment instructions, enhancing alignment between model behavior and human intents without fine-tuning. The method is lightweight, resource-efficient, and compatible with a wide range of open-source LLMs. It can function independently during inference or be integrated with training-based alignment strategies. Moreover, SDA supports personalized preference alignment, enabling flexible control over the model response behavior. Empirical results demonstrate that SDA consistently improves alignment performance across 8 open-source LLMs with varying scales and diverse origins, evaluated on three key alignment dimensions, helpfulness, harmlessness, and honesty (3H). Specifically, SDA achieves average gains of 64.4% in helpfulness, 30% in honesty and 11.5% in harmlessness across the tested models, indicating its effectiveness and generalization across diverse models and application scenarios.

大模型对齐无训练优化推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。