提出新型双向门控网络,高效实现嵌入式语言去噪。
DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
- 设计噪声自适应的双向门控结构,替代传统二次复杂度注意力。
- 在16k序列下实现2.72倍吞吐速度提升,困惑度降至21.228。
- 适合资源受限场景下的连续语言去噪任务,尤其关注效率与稳定性。
嵌入式语言流(ELF)主要依赖全非因果注意力进行迭代去噪,在每个采样步骤中重复产生二次序列混合开销。门控增量网络(GDN)提供了高效的循环替代方案,但其标准因果形式无法直接捕捉ELF所需的双向上下文。本文提出DeltaFlow,一种用于连续语言去噪的噪声自适应双向GDN主干。研究了两种变体:DeltaFlow-A在层间交替扫描方向,DeltaFlow-P在每层内并行执行前向与后向扫描。进一步引入噪声自适应记忆控制和预定时间状态一致性(TSC),以稳定相邻噪声水平下的隐藏表示。在OpenWebText数据集上,使用32步随机微分方程采样器,DeltaFlow-P将生成困惑度从全注意力基线的24.218降至21.228,同时保持相近的单字熵,训练令牌暴露量为360亿,低于基线的450亿。在仅去噪器的基准测试中,于16k序列长度下,DeltaFlow-P相比全注意力基线实现2.72倍的吞吐加速。结果表明,DeltaFlow是高效连续语言去噪中密集注意力的有力替代方案。
原文摘要 · Abstract (English)
Embedded Language Flows (ELF) rely primarily on full non-causal attention for iterative denoising, repeatedly incurring quadratic sequence-mixing cost at each sampling step. Gated Delta Networks (GDNs) provide an efficient recurrent alternative, but their standard causal formulation cannot directly capture the bidirectional context required by ELF. We introduce DeltaFlow, a noise-adaptive bidirectional GDN backbone for continuous language denoising. We study two variants: DeltaFlow-A, which alternates scan directions across layers, and DeltaFlow-P, which performs parallel forward and backward scans within each layer. We further introduce noise-adaptive memory control and scheduled Temporal State Consistency (TSC) to stabilize hidden representations across nearby noise levels. On OpenWebText, using a 32-step stochastic differential equation sampler, DeltaFlow-P reduces generated perplexity from 24.218 for the full-attention ELF baseline to 21.228 while maintaining comparable unigram entropy, with 36B training-token exposure compared with 45B for the baseline. In a denoiser-only benchmark, DeltaFlow-P achieves a 2.72x throughput speedup over the full-attention baseline at a sequence length of 16k. These results show that DeltaFlow is a promising alternative to dense attention for efficient continuous language denoising.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。