arXiv:2506.01966cs.LGcs.AI2025-06

统一用稀疏矩阵表示卷积、循环和注意力,让不同模型共享数学基础。

Unified Sparse-Matrix Representations for Diverse Neural Architectures

  • 将卷积、递归、注意力统一为稀疏矩阵乘法,通过不同三角结构实现。
  • 在多个数据集上性能媲美甚至超过原生模型,收敛速度更快。
  • 适合研究模型统一理论或追求高效硬件部署的开发者。

深度神经网络在视觉、序列和语言任务中采用专用架构,但这种多样化掩盖了其内在共性。本文提出一种统一的矩阵序框架,将卷积、循环和自注意力操作均表示为稀疏矩阵乘法:卷积通过上三角权重矩阵实现一阶变换;递归由下三角矩阵编码逐步更新;注意力则自然表现为三阶张量分解。在温和假设下,证明其与标准CNN、RNN和Transformer层代数同构。在图像分类(MNIST、CIFAR-10/100、Tiny ImageNet)、时间序列预测(ETTh1、Electricity Load Diagrams)及语言建模/分类(AG News、WikiText-2、Penn Treebank)上的实证表明,稀疏矩阵形式化方法在性能上可匹配或超越原生模型,且收敛于相当或更少轮次。该框架将架构设计简化为稀疏模式选择,契合GPU并行特性,并可利用成熟的代数优化工具。本工作为多元神经架构建立严格的数学基础,开辟了可解释、面向硬件的网络设计新路径。

原文摘要 · Abstract (English)

Deep neural networks employ specialized architectures for vision, sequential and language tasks, yet this proliferation obscures their underlying commonalities. We introduce a unified matrix-order framework that casts convolutional, recurrent and self-attention operations as sparse matrix multiplications. Convolution is realized via an upper-triangular weight matrix performing first-order transformations; recurrence emerges from a lower-triangular matrix encoding stepwise updates; attention arises naturally as a third-order tensor factorization. We prove algebraic isomorphism with standard CNN, RNN and Transformer layers under mild assumptions. Empirical evaluations on image classification (MNIST, CIFAR-10/100, Tiny ImageNet), time-series forecasting (ETTh1, Electricity Load Diagrams) and language modeling/classification (AG News, WikiText-2, Penn Treebank) confirm that sparse-matrix formulations match or exceed native model performance while converging in comparable or fewer epochs. By reducing architecture design to sparse pattern selection, our matrix perspective aligns with GPU parallelism and leverages mature algebraic optimization tools. This work establishes a mathematically rigorous substrate for diverse neural architectures and opens avenues for principled, hardware-aware network design.

神经网络稀疏矩阵统一架构代数优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。