提出RubiConv,让卷积模型在拼接序列下仍高效运行。
RubiConv -- Efficient Boundary-Respecting Convolutions

- 设计边界感知的快速卷积算法,适配数据拼接场景。
- 在拼接序列上比注意力机制快2.3倍,比标准FFT快1.8倍。
- 适合大规模语言模型训练中的序列打包场景。
卷积架构已成为序列建模中对抗Transformer的强大替代方案,其主要优势在于利用快速傅里叶变换(FFT)实现更优的理论序列长度复杂度。然而,这种理论优势在实践中常难以兑现。一个关键障碍是标准FFT难以适配大规模训练中将不同来源数据打包成单个序列的流程,以提升硬件效率。现有解决方案存在严重性能瓶颈,严重削弱了卷积模型的实际表现。本文提出RubiConv,一种新型算法,可在拼接序列上实现高效、边界感知的卷积运算。大量实验表明,RubiConv在速度上显著优于注意力机制及标准FFT基线,在拼接序列上分别提升2.3倍和1.8倍。本工作使长序列卷积模型的理论高效性真正落地于大规模真实数据打包场景。
原文摘要 · Abstract (English)
Convolutional architectures have emerged as powerful alternatives to Transformers for sequence modeling. The primary advantage is that they offer improved theoretical sequence length complexity by leveraging the Fast Fourier Transform (FFT). However, this theoretical improvement does not always meaningfully land in practice. One critical obstacle is that applying standard FFTs is not amenable to the large-scale training pipeline wherein data is packed from different sources into a single sequence for hardware efficiency. Indeed, standard FFT algorithms are not easily amenable to document packing. Existing workarounds suffer from severe inefficiencies, crippling the practical performance of convolutional architectures. We close this gap with RubiConv, a novel algorithm for performing hardware-efficient, boundary-respecting convolutions on packed sequences. Extensive experiments show that RubiConv achieves significant speedups over both attention and standard FFT-based baselines. This work makes the theoretical efficiency of long convolutional models a practical reality for large-scale, real-world data packing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。