arXiv:2504.08801cs.LGcs.CL2025-04被引 7

用可学习的小波变换替代注意力,高效建模长序列。

Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention

  • 以可学习的多尺度哈尔小波替换自注意力机制
  • 在WMT16英德翻译任务上达到与原模型相当的BLEU分数
  • 计算复杂度线性增长,适合长序列和资源受限场景

Transformer架构依赖自注意力机制,在自然语言处理任务中通过建模长距离依赖实现了顶尖性能。然而,自注意力的计算复杂度随序列长度呈二次增长,给处理超长序列或资源受限环境带来挑战。本文提出可学习多尺度小波Transformer(LMWT),将标准点积自注意力替换为可学习的多尺度哈尔小波变换模块。利用小波固有的多分辨率特性,LMWT能高效捕捉局部细节与全局上下文。关键在于,小波变换的参数(包括各尺度系数)在训练中端到端学习,使模型能根据数据和任务自适应分解策略。我们给出了可学习哈尔小波模块的数学推导及其在Transformer中的集成方式,并提供架构图。在标准机器翻译基准WMT16 En-De上进行了全面实验,对比了基于BLEU、困惑度和词元准确率的基线自注意力Transformer。此外,分析了计算复杂度,突出本方法线性增长的优势;讨论了其在相关工作中的创新性,并通过可视化学习到的哈尔系数探讨了模型可解释性。结果表明,LMWT在保持竞争力性能的同时,显著降低计算开销,是高效序列建模的有前景新选择。

原文摘要 · Abstract (English)

Transformer architectures, underpinned by the self-attention mechanism, have achieved state-of-the-art results across numerous natural language processing (NLP) tasks by effectively modeling long-range dependencies. However, the computational complexity of self-attention, scaling quadratically with input sequence length, presents significant challenges for processing very long sequences or operating under resource constraints. This paper introduces the Learnable Multi-Scale Wavelet Transformer (LMWT), a novel architecture that replaces the standard dot-product self-attention with a learnable multi-scale Haar wavelet transform module. Leveraging the intrinsic multi-resolution properties of wavelets, the LMWT efficiently captures both local details and global context. Crucially, the parameters of the wavelet transform, including scale-specific coefficients, are learned end-to-end during training, allowing the model to adapt its decomposition strategy to the data and task. We present the detailed mathematical formulation of the learnable Haar wavelet module and its integration into the transformer framework, supplemented by an architectural diagram. We conduct a comprehensive experimental evaluation on a standard machine translation benchmark (WMT16 En-De), comparing the LMWT against a baseline self-attention transformer using metrics like BLEU score, perplexity, and token accuracy. Furthermore, we analyze the computational complexity, highlighting the linear scaling of our approach, discuss its novelty in the context of related work, and explore the interpretability offered by visualizing the learned Haar coefficients. Our results indicate that the LMWT achieves competitive performance while offering substantial computational advantages, positioning it as a promising and novel alternative for efficient sequence modeling.

小波变换Transformer序列建模高效计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。