根据注意力依赖关系自动划分文本块,提升扩散语言模型的生成效率与准确性。
GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models
- 基于注意力机制分析跨标记依赖,动态识别可并行处理的语义区域。
- 在多个基准上显著提升块扩散模型的生成准确率,计算开销极小。
- 无需训练即可集成,适合追求高效高质生成的开发者使用。
块扩散技术可在扩散语言模型中实现高效的并行精炼,但其解码行为对块大小极为敏感。现有块大小策略依赖固定规则或启发式信号,未考虑决定哪些标记可安全联合精炼的依赖几何结构。为此,我们提出一种几何视角的扩散解码:具有强因果顺序的区域需串行更新,而语义连贯的区域则可并行精炼。我们引入GeoBlock,一种基于依赖几何的块粒度推断框架,直接从注意力衍生的依赖几何中确定块粒度。相比预设调度或局部置信度启发式,GeoBlock分析跨标记依赖模式,识别几何稳定的精炼区域,并在解码过程中动态确定合适的块边界。通过适应依赖几何结构,GeoBlock在保持块扩散并行效率的同时,确保依赖一致性,表现出类似自回归的可靠性。该方法无需额外训练,可无缝集成至现有块扩散架构。大量实验表明,GeoBlock能可靠识别几何一致的块边界,并在仅增加少量计算成本的情况下提升块扩散的准确性。
原文摘要 · Abstract (English)
Block diffusion enables efficient parallel refinement in diffusion language models, but its decoding behavior depends critically on block size. Existing block-sizing strategies rely on fixed rules or heuristic signals and do not account for the dependency geometry that determines which tokens can be safely refined together. This motivates a geometry view of diffusion decoding: \emph{regions with strong causal ordering require sequential updates, whereas semantically cohesive regions admit parallel refinement.} We introduce GeoBlock, a geometry-aware block inference framework that determines block granularity directly from attention-derived dependency geometry. Instead of relying on predefined schedules or local confidence heuristics, GeoBlock analyzes cross-token dependency patterns to identify geometrically stable refinement regions and dynamically determines appropriate block boundaries during decoding. By adapting block granularity to the dependency geometry, GeoBlock preserves the parallel efficiency of block diffusion while enforcing dependency-consistent refinement that exhibits autoregressive reliability. GeoBlock requires no additional training and integrates seamlessly into existing block diffusion architectures. Extensive experiments across multiple benchmarks show that GeoBlock reliably identifies geometry-consistent block boundaries and improves the accuracy of block diffusion with only a small additional computational budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。