用波动方程解耦频率与时间,实现高效全局视觉建模。
WaveFormer: Frequency-Time Decoupled Vision Modeling with Wave Equation

- 将特征图视为受波动方程支配的信号,显式建模频率与传播时间关系
- 在图像分类等任务上达到主流ViT性能,推理速度提升1.6倍,计算量减少30%
- 适合追求高吞吐、低计算开销的视觉模型部署场景
视觉建模虽因Transformer快速发展,但其注意力机制缺乏对语义信息空间传播的理论基础。本文从波动视角重新审视该问题:将特征图视为随内部传播时间(与网络深度对齐)演化的空间信号,其演化由欠阻尼波动方程控制。该框架显式建模从低频全局布局到高频边缘纹理的空间频率,并调控其与传播时间的交互。我们推导出闭式解,提出波传播算子(WPO),作为轻量模块实现全局交互,计算复杂度仅为O(N log N),远低于注意力机制。基于WPO,我们构建了可直接替换标准ViTs和CNNs的WaveFormer系列模型,在图像分类、目标检测与语义分割任务中表现优异,相较注意力模型最高提升1.6倍吞吐,减少30% FLOPs。结果表明,波动传播引入了与热扩散方法互补的建模偏置,能有效捕捉全局连贯性与高频细节,对丰富视觉语义至关重要。代码已开源:https://github.com/ZishanShu/WaveFormer。
原文摘要 · Abstract (English)
Vision modeling has advanced rapidly with Transformers, whose attention mechanisms capture visual dependencies but lack a principled account of how semantic information propagates spatially. We revisit this problem from a wave-based perspective: feature maps are treated as spatial signals whose evolution over an internal propagation time (aligned with network depth) is governed by an underdamped wave equation. In this formulation, spatial frequency-from low-frequency global layout to high-frequency edges and textures-is modeled explicitly, and its interaction with propagation time is controlled rather than implicitly fixed. We derive a closed-form, frequency-time decoupled solution and implement it as the Wave Propagation Operator (WPO), a lightweight module that models global interactions in O(N log N) time-far lower than attention. Building on WPO, we propose a family of WaveFormer models as drop-in replacements for standard ViTs and CNNs, achieving competitive accuracy across image classification, object detection, and semantic segmentation, while delivering up to 1.6x higher throughput and 30% fewer FLOPs than attention-based alternatives. Furthermore, our results demonstrate that wave propagation introduces a complementary modeling bias to heat-based methods, effectively capturing both global coherence and high-frequency details essential for rich visual semantics. Codes are available at: https://github.com/ZishanShu/WaveFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。