小模型先写开头，大模型续写，提升低资源对齐效果
Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
- 用小模型生成高质量开头，大模型接力完成，自动切换
- 在GenerAlign数据集上训练的小模型使大模型性能超越所有基线
- 避免下游任务性能下降，适合资源有限但需高对齐的场景
大型语言模型需与人类偏好对齐，以避免生成攻击性、虚假或无意义内容。尽管低资源对齐方法日益流行，仍面临生成高质量且对齐内容的挑战。受生成对齐响应的难度集中于解码初期的启发，我们提出一种新框架——弱到强解码（WSD），通过一个小型对齐模型引导基础模型增强对齐能力。小模型首先生成高度对齐的开头部分，随后由大模型继续生成，由精心设计的自动切换机制控制。我们还构建了新数据集GenerAlign，用于微调小型Pilot-3B作为草稿模型。该模型在WSD框架下显著提升多种基础模型的表现，优于所有基线方法，且未引发下游任务性能下降（即避免对齐税）。通过大量实验，我们验证了不同设置的影响、时间效率，并深入分析了WSD的内在机制。
原文摘要 · Abstract (English)
Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。