用自回归模型做老师，让扩散模型学会对齐偏好。
AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?
- 用自回归模型的权重知识引导扩散模型对齐。
- 在多任务测试中平均得分达69.08%，表现优异。
- 无需复杂训练，适合希望快速提升扩散模型对齐能力的研究者。
扩散型大语言模型（DLLMs）作为自回归模型的有力替代，支持多位置并行生成 token。然而，由于基于证据下界（ELBO）的似然估计引入高方差，其偏好对齐仍具挑战性。本文提出 AR-MAP，一种新型迁移学习框架，利用偏好对齐的自回归大语言模型（AR-LLMs）作为隐式教师，指导 DLLM 对齐。我们发现，通过简单的权重缩放，DLLMs 可有效吸收来自 AR-LLMs 的对齐知识，充分利用两类生成范式间的共享架构结构。关键在于，该方法避免了直接对齐带来的高方差与计算开销。在多个偏好对齐任务上的全面实验表明，AR-MAP 在性能上达到或超越现有针对 DLLM 的对齐方法，所有任务与模型的平均得分为 69.08%。代码已开源：https://github.com/AMAP-ML/AR-MAP。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) have emerged as a powerful alternative to autoregressive models, enabling parallel token generation across multiple positions. However, preference alignment of DLLMs remains challenging due to high variance introduced by Evidence Lower Bound (ELBO)-based likelihood estimation. In this work, we propose AR-MAP, a novel transfer learning framework that leverages preference-aligned autoregressive LLMs (AR-LLMs) as implicit teachers for DLLM alignment. We reveal that DLLMs can effectively absorb alignment knowledge from AR-LLMs through simple weight scaling, exploiting the shared architectural structure between these divergent generation paradigms. Crucially, our approach circumvents the high variance and computational overhead of direct DLLM alignment and comprehensive experiments across diverse preference alignment tasks demonstrate that AR-MAP achieves competitive or superior performance compared to existing DLLM-specific alignment methods, achieving 69.08\% average score across all tasks and models. Our Code is available at https://github.com/AMAP-ML/AR-MAP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。