让语言模型在训练和推理时都能灵活调整对齐程度，提升效率与性能。
Flexible Realignment of Language Models
- 训练时通过融合参考模型与已对齐模型的输出，实现可控对齐。
- 训练中减少54.63%的令牌使用量，且性能不下降，优于基线33.86%。
- 推理时引入层适配器，支持快速/深度思考切换，适合需灵活响应场景。
当语言模型无法达到预期性能时，需进行重对齐。我们提出一种灵活的重对齐框架，可在训练和推理阶段实现对齐程度的定量控制。该框架包含训练时重对齐（TrRa），通过可控融合参考模型与已对齐模型的logits，高效重对齐参考模型。例如，TrRa在DeepSeek-R1-Distill-Qwen-1.5B上将令牌使用量降低54.63%，无性能损失，优于DeepScaleR-1.5B的33.86%。为补充推理阶段，我们引入层适配器，实现推理时重对齐（InRa）。该适配器初始为恒等变换，插入原模型层前。推理时，输入嵌入同时经适配器与原层处理，后续层继续运行，并在logit层进行可控插值。我们将DeepSeek-R1-Distill-Qwen-7B从慢思考模型升级为支持快慢思考的模型，实现推理时灵活对齐控制，甚至因促进深层推理而超越原始性能。
原文摘要 · Abstract (English)
Realignment becomes necessary when a language model (LM) fails to meet expected performance. We propose a flexible realignment framework that supports quantitative control of alignment degree during training and inference. This framework incorporates Training-time Realignment (TrRa), which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by 54.63% on DeepSeek-R1-Distill-Qwen-1.5B without any performance degradation, outperforming DeepScaleR-1.5B's 33.86%. To complement TrRa during inference, we introduce a layer adapter that enables smooth Inference-time Realignment (InRa). This adapter is initialized to perform an identity transformation at the bottom layer and is inserted preceding the original layers. During inference, input embeddings are simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit level. We upgraded DeepSeek-R1-Distill-Qwen-7B from a slow-thinking model to one that supports both fast and slow thinking, allowing flexible alignment control even during inference. By encouraging deeper reasoning, it even surpassed its original performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。