arXiv:2606.19538cs.AIcs.LG2026-06

一个模型统一了卷积、注意力和循环网络,从数据中自动学习最优处理方式。

ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

  • 用可学习的积分核统一建模信号交互,融合三类主流架构的核心机制。
  • 在多个基准上性能媲美甚至超越专用模型,涵盖图像、文本、点云等任务。
  • 适合希望减少模型种类、追求统一框架的研究者或工程团队使用。

卷积网络、循环网络和变压器各自体现了局部性、序列记忆和内容依赖的成对交互等归纳偏置,自诞生以来数学上始终分离。我们发现这种分裂并非信号处理的根本差异,而是对单一潜在数学对象认识不全所致:一个可学习的积分变换。我们提出积分变换网络(ITNet),其核心是一个同时依赖位置与特征的可学习核函数,该核由小型MLP实现,能动态建模成对交互,使模型从数据中自适应行为。我们证明,卷积、自注意力(含多头)以及自回归循环(含LSTM、GRU、S4、Mamba)均可作为特定参数化下的特例,且ITNet是连续算子的通用逼近器。为实用化,我们设计分块核融合、重要性加权蒙特卡洛积分和可学习低秩分解,实现高效可扩展计算。单个共享算子搭配轻量调制编码器,在ImageNet-1K、GLUE、ModelNet40、VQA v2和NLVR2上表现匹配或优于专用基线模型。结果表明,单一可学习交互机制可从数据中还原三类架构的行为。

原文摘要 · Abstract (English)

Convolutional networks, recurrent networks, and transformers each encode different inductive biases -- locality, sequential memory, and content-dependent pairwise interaction -- and have remained mathematically distinct since their inception. We show that this fragmentation reflects not a fundamental diversity in how signals should be processed, but rather incomplete views of a single underlying mathematical object: a learnable integral transform. We introduce the Integral Transform Network (ITNet), a unified architecture built around a learnable kernel that depends jointly on positions and features. This kernel is implemented as a small neural network, specifically an MLP, that models pairwise interactions, enabling the model to adapt its behavior from data. We show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as special cases under appropriate parameterizations, and that ITNet is a universal approximator of continuous operators. To make this practical, we develop tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization, enabling efficient and scalable computation. A single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on ImageNet-1K , GLUE, ModelNet40, VQA\,v2 and NLVR2. The results demonstrate that a single learned interaction mechanism can recover the behavior of all three architectural families from data.

统一模型积分变换架构融合可学习核

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。