让语言模型自由调节对齐程度，像调音量一样控制输出风格。
Adding Alignment Control to Language Models
- 在模型前加一层身份层，仅微调该层实现对齐控制。
- 只需调节一个系数，就能在完全对齐和原始输出间平滑切换。
- 适合需要灵活控制输出风格的场景，如内容创作或合规审查。
后训练对齐已成为提升语言模型可用性的关键因素。然而，对齐强度因人而异。本文提出一种将对齐控制融入单个模型的方法，称为CLM。该方法在初始层前增加一层身份层，仅对该层进行偏好学习，将未对齐的输入词嵌入映射到对齐空间。实验表明，这种高效微调方法性能接近全模型微调。推理时，输入嵌入通过对齐与非对齐层处理，再通过插值系数融合。通过调节该参数，对齐效果呈现清晰的插值与外推现象。
原文摘要 · Abstract (English)
Post-training alignment has increasingly become a crucial factor in enhancing the usability of language models (LMs). However, the strength of alignment varies depending on individual preferences. This paper proposes a method to incorporate alignment control into a single model, referred to as CLM. This approach adds one identity layer preceding the initial layers and performs preference learning only on this layer to map unaligned input token embeddings into the aligned space. Experimental results demonstrate that this efficient fine-tuning method performs comparable to full fine-tuning. During inference, the input embeddings are processed through the aligned and unaligned layers, which are then merged through the interpolation coefficient. By controlling this parameter, the alignment exhibits a clear interpolation and extrapolation phenomenon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。