一个模型统一了卷积、注意力和循环网络,从数据中自动学习最优处理方式。
ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence
- 用可学习的积分核统一建模信号交互,融合三类主流架构的核心机制。
- 在多个基准上性能媲美甚至超越专用模型,涵盖图像、文本、点云等任务。
- 适合希望减少模型种类、追求统一框架的研究者或工程团队使用。
卷积网络、循环网络和变压器各自体现了局部性、序列记忆和内容依赖的成对交互等归纳偏置,自诞生以来数学上始终分离。我们发现这种分裂并非信号处理的根本差异,而是对单一潜在数学对象认识不全所致:一个可学习的积分变换。我们提出积分变换网络(ITNet),其核心是一个同时依赖位置与特征的可学习核函数,该核由小型MLP实现,能动态建模成对交互,使模型从数据中自适应行为。我们证明,卷积、自注意力(含多头)以及自回归循环(含LSTM、GRU、S4、Mamba)均可作为特定参数化下的特例,且ITNet是连续算子的通用逼近器。为实用化,我们设计分块核融合、重要性加权蒙特卡洛积分和可学习低秩分解,实现高效可扩展计算。单个共享算子搭配轻量调制编码器,在ImageNet-1K、GLUE、ModelNet40、VQA v2和NLVR2上表现匹配或优于专用基线模型。结果表明,单一可学习交互机制可从数据中还原三类架构的行为。
原文摘要 · Abstract (English)
Convolutional networks, recurrent networks, and transformers each encode different inductive biases -- locality, sequential memory, and content-dependent pairwise interaction -- and have remained mathematically distinct since their inception. We show that this fragmentation reflects not a fundamental diversity in how signals should be processed, but rather incomplete views of a single underlying mathematical object: a learnable integral transform. We introduce the Integral Transform Network (ITNet), a unified architecture built around a learnable kernel that depends jointly on positions and features. This kernel is implemented as a small neural network, specifically an MLP, that models pairwise interactions, enabling the model to adapt its behavior from data. We show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as special cases under appropriate parameterizations, and that ITNet is a universal approximator of continuous operators. To make this practical, we develop tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization, enabling efficient and scalable computation. A single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on ImageNet-1K , GLUE, ModelNet40, VQA\,v2 and NLVR2. The results demonstrate that a single learned interaction mechanism can recover the behavior of all three architectural families from data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。