arXiv:2601.22954cs.CLcs.AI2026-01被引 8

让丢弃的解码信息重用,提升扩散语言模型效率与准确率。

Residual Context Diffusion Language Models

  • 将被丢弃的令牌表示转为上下文残差,注入后续去噪步骤
  • 在多个基准上提升4-11个百分点准确率,挑战任务上接近翻倍
  • 仅需约3亿训练数据即可改造现有模型,计算开销极低

扩散语言模型(dLLMs)因其可并行生成多词而成为自回归模型的有力替代。然而,当前主流的分块式dLLMs依赖“重掩码”机制,仅保留最置信的词汇,其余信息被丢弃,造成计算浪费。本文发现,这些被丢弃的令牌仍携带有用上下文信息,可用于后续解码。为此提出残差上下文扩散(RCD)模块,将废弃令牌表征转化为上下文残差,并回注入下一去噪步骤。RCD采用解耦的两阶段训练流程,避免反向传播带来的内存瓶颈。在长链思维(SDAR)和短链指令遵循(LLaDA)任务上验证,仅需约3亿条数据即可将标准dLLM转换为RCD范式。RCD在多种基准上稳定提升4-11个百分点准确率,额外计算开销极小。尤其在高难度AIME任务中,准确率近乎翻倍,达到基线峰值精度时所需去噪步数减少4-5倍。

原文摘要 · Abstract (English)

Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a "remasking" mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ~300 million tokens. RCD consistently improves frontier dLLMs by 4-11 percentage points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4-5x fewer denoising steps at baseline's peak accuracy.

扩散模型语言模型高效推理上下文重用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。