解决扩散语言模型块边界错误传播问题,提升生成连贯性。
Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models

- 提出多块编辑机制,用跨块上下文修正块边界处的生成结果。
- 在12个基准上平均提升至64.24分,AIME 2025最高增13.33点。
- 无需训练即可应用,保持96.7%以上标准解码吞吐量,适合部署优化。
块扩散是扩展离散扩散语言模型(dLLMs)的主流方法,固定块大小在保持并行解码的同时将二次注意力开销控制在可接受范围。然而,块式生成存在结构性缺陷:靠近块边界的词元缺乏后续块的上下文,且已定稿块中的错误会成为后续生成的不可逆上下文。我们称之为块边界问题。对比有无后块上下文的预测发现,边界敏感度急剧上升:在AIME 2025上,块末四分之一区域的平均自包含性偏差(SCD)是开头四分之一的61.3倍。为此,我们提出多块编辑(MBE),利用跨块上下文修正已解码词元。训练无关的MBE通过开放完整注意力窗口实现,无需参数更新。为缓解块扩散训练与MBE推理间的不匹配,多块编辑SFT引入双向注意力掩码并逐步扩大编辑跨度。同时,我们扩展SGLang,引入多形状CUDA图池和细粒度KV缓存控制,以支持高效变长编辑。在LLaDA2.1-Mini上对12个基准的实验表明,改进效果广泛而稳定:训练无关的MBE在所有基准上均优于或匹配标准解码;完整版MBE使12个基准平均得分从61.45提升至64.24,其中在AIME 2025上最高提升13.33分,同时在四个数据集上保持87.3%–96.7%的标准解码端到端吞吐率。
原文摘要 · Abstract (English)
Block diffusion is the dominant approach for scaling discrete diffusion language models (dLLMs), as fixed-size blocks preserve parallel decoding while keeping quadratic attention costs tractable. Yet blockwise generation creates a structural weakness: tokens near a block boundary lack future cross-block context, and errors in finalized blocks become irreversible context for later generation. We call this the block boundary problem. Measuring predictions with and without later-block context shows that boundary sensitivity rises sharply: on AIME 2025, mean self-containedness divergence (SCD) in the last quarter of a block is 61.3 times that in the first quarter. We propose Multi-Block Editing (MBE), which revises decoded tokens using cross-block context. Training-Free MBE reopens a full-attention window over selected blocks without parameter updates. To address the mismatch between block-diffusion training and MBE inference, Multi-Block Edit SFT introduces bidirectional attention masks and progressively enlarges the editing span. We also extend SGLang with a multi-shape CUDA Graph pool and fine-grained KV-cache control for efficient variable-length editing. Experiments on LLaDA2.1-Mini across 12 benchmarks show broad, consistent gains. Training-Free MBE improves or matches standard decoding on every benchmark. Full MBE raises the 12-benchmark average from 61.45 to 64.24, with gains of up to 13.33 points on AIME 2025, while retaining 87.3--96.7% of standard-decoding end-to-end throughput across four datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。