推理阶段提升大模型对齐效果，无需额外训练
Alignment-Aware Decoding
- 在推理时动态调整生成策略以增强对齐
- 跨多种模型规模和基准均优于现有方法
- 数据少时可生成高质量合成数据，适合资源受限场景
大语言模型的对齐问题仍是自然语言处理的核心挑战。偏好优化已成为一种流行且有效的方法，通常通过训练阶段或提示工程实现。本文提出推理阶段对齐增强（Alignment-Aware Decoding, AAD），直接在推理时提升模型对齐能力。理论上，AAD可视为隐式奖励优化，但无需额外训练，仅需标准DPO设置即可生效。实验表明，AAD在多个对齐基准和不同模型规模下持续优于强基线。此外，在数据受限情况下，AAD能生成高质量合成数据，进一步提升标准解码下的对齐性能，为标注数据稀缺场景提供实用解决方案。
原文摘要 · Abstract (English)
Alignment of large language models remains a central challenge in natural language processing. Preference optimization has emerged as a popular and effective method for improving alignment, typically through training-time or prompt-based interventions. In this paper, we introduce alignment-aware decoding (AAD), a method to enhance model alignment directly at inference. Theoretically, AAD can be interpreted as implicit reward optimization, yet it requires no specialized training beyond the standard DPO setup. Empirically, AAD consistently outperforms strong baselines across diverse alignment benchmarks and model scales. Moreover, in data-constrained settings, AAD can produce high-quality synthetic data to improve alignment under standard decoding, providing a practical solution when labeled data is limited.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。