将Transformer转换为有向图神经网络，提升效率与性能。
Converting Transformers into DGNNs Form
- 用有向图傅里叶变换构造新型卷积单元替代自注意力
- 在长文本分类和DNA序列分类任务中表现更优且计算高效
- 适合追求轻量化高性能模型的开发者使用
深度学习的进展使Transformer架构成为主流建模范式。其成功核心在于自注意力机制，该机制通过查询与键矩阵的相似性评分来调制值矩阵。这一操作与有向图卷积高度相似，促使我们探究有向图卷积是否可作为自注意力的替代方案。本研究通过引入基于有向图傅里叶变换的合成酉有向图卷积，形式化了该思想。由此产生的模型称为Converter，可将Transformer有效转化为有向图神经网络（DGNN）形式。我们在Long-Range Arena基准、长文档分类及基于DNA序列的分类任务上测试Converter。实验结果表明，Converter在保持计算效率与结构简洁性的同时，实现了更优性能，确立了其作为轻量但强大的Transformer变体的地位。
原文摘要 · Abstract (English)
Recent advances in deep learning have established Transformer architectures as the predominant modeling paradigm. Central to the success of Transformers is the self-attention mechanism, which scores the similarity between query and key matrices to modulate a value matrix. This operation bears striking similarities to digraph convolution, prompting an investigation into whether digraph convolution could serve as an alternative to self-attention. In this study, we formalize this concept by introducing a synthetic unitary digraph convolution based on the digraph Fourier transform. The resulting model, which we term Converter, effectively converts a Transformer into a Directed Graph Neural Network (DGNN) form. We have tested Converter on Long-Range Arena benchmark, long document classification, and DNA sequence-based taxonomy classification. Our experimental results demonstrate that Converter achieves superior performance while maintaining computational efficiency and architectural simplicity, which establishes it as a lightweight yet powerful Transformer variant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。