通过频域引导实现扩散语言模型的非自回归高效生成
FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
- 基于频域分析，用低频分量捕捉全局结构，高频分量刻画局部细节
- 在LLaDA和SDAR上分别提升20.4%和16.0%，超越同规模自回归模型
- 适合追求高速生成且需高质量输出的部署场景
尽管扩散语言模型（dLLMs）具备非自回归潜力，现有解码策略仍存在位置偏差，难以充分发挥任意生成能力。本文首次开展dLLMs的频域分析，发现隐藏状态中的低频成分主要编码全局结构与长程依赖，高频成分则负责表征局部细节。基于此，提出FourierSampler，通过频域滑动窗口机制动态引导模型实现‘由结构到细节’的生成流程。该方法在LLaDA和SDAR基准上分别取得20.4%（LLaDA1.5-8B）和16.0%（LLaDA-8B-Instruct）的相对提升，显著优于同规模自回归模型如Llama3.1-8B-Instruct。
原文摘要 · Abstract (English)
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。