提出新型非对称双向上下文模型,实现并行生成的高速与高质量兼顾。
Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

- 采用因果注意力+轻量反向Mamba结构,兼顾缓存效率与右端上下文
- 在批量推理中速度比双向扩散模型快2.4至12.9倍,比自回归模型快1.9至2.9倍
- 适合追求高吞吐且需保留部分右端信息的实时生成场景
离散扩散语言模型(dLLMs)可并行恢复被掩码的词元,显著提升生成速度。然而,该框架面临根本性设计矛盾:采用双向注意力虽能获得强生成质量,但无法兼容键值缓存,限制批量服务下的推理吞吐;而因果注意力虽支持高效缓存,却丢失全部右侧上下文,严重降低生成质量。本文提出双焦点dLLMs新范式,通过非对称双向上下文解决此矛盾。具体实现为R2LM(Right-to-Left Mamba),结合标准因果注意力(提供精确左端上下文,完全兼容KV缓存)与轻量级反向Mamba状态空间模型(无损提供压缩右端上下文)。在60B token持续预训练的Qwen3-1.7B上实验表明,R2LM在批量服务中相比双向dLLMs吞吐提升2.4至12.9倍,比自回归基线快1.9至2.9倍,同时在多数基准上超越因果基线,在平均性能上超过双向dLLM。
原文摘要 · Abstract (English)
Discrete diffusion language models (dLLMs) recover masked tokens in parallel, offering significant speedups over autoregressive (AR) generation. However, such promising frameworks face a fundamental architectural design dilemma: \ding{182} Adopting bidirectional attention achieves strong generation quality by allowing each position to access the full context, but is inherently incompatible with KV caching, limiting inference throughput in batch-serving scenarios; \ding{183} Conversely, causal attention enables efficient cached inference but loses all right-side context, substantially degrading generation quality. This paper introduces Bifocal dLLMs, a new paradigm that resolves this dilemma through \emph{asymmetric bidirectional context}. Analogous to bifocal lenses, we instantiate the paradigm as \textbf{R2LM} (Right-to-Left Mamba), which combines two complementary mechanisms: $a$) standard causal attention providing precise left-context with full KV cache compatibility, while $b$) a lightweight reverse Mamba SSM sidecar supplying compressed right-side context without breaking cacheability. Comprehensive experiments on continued pretraining of Qwen3-1.7B with 60B tokens demonstrate that R2LM achieves $2.4\times$ to $12.9\times$ higher throughput than bidirectional dLLMs and $1.9\times$ to $2.9\times$ speedup over AR baselines in batch serving through parallel decoding with KV caching, while exceeding the causal baseline on most benchmarks and surpassing the bidirectional dLLM on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。