提出可直接处理多维数据的高效注意力替代方法，兼顾全局感知与结构保持。
Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions
- 用隐式参数化卷积核实现多维数据的全局输入依赖计算
- 在基因组、医学影像等任务中精度媲美主流注意力模型
- 适合需要长序列建模且保持空间结构的多维数据场景
传统注意力的子二次替代方案在处理多维数据时存在妥协：标准卷积缺乏全局感受野且不具输入依赖性，而循环模型需将图像、体数据及偏微分方程（PDE）等强行转为一维扫描顺序，破坏其空间结构。本文提出 extit{HyenaND}，一种子二次、全局、输入依赖的算子，通过隐式参数化的多维卷积核直接作用于多维数据的原始几何结构。其CUDA实现 exttt{nSubQ} 融合FFT卷积路径，将 HyenaND 的 $/mathcal{O}(L /log L)$ 复杂度转化为实际运行时加速。在长序列基因组、计算机视觉、医学影像和PDE建模任务中，纯 HyenaND 模型堆叠的精度达到强注意力基线水平；混合配置（交错使用 HyenaND 与注意力层）则超越纯注意力与强递归混合模型。
原文摘要 · Abstract (English)
Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc $1\rm D$ scan order that violates their spatial structure. We introduce \textit{HyenaND}, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, \texttt{nSubQ}, fuses the FFT-convolution path to turn HyenaND's $\mathcal{O}(L \log L)$ scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。