arXiv:2508.19167cs.CV2025-08

用椭圆函数设计新位置编码,让视觉Transformer更好理解图像空间结构。

Beyond flattening: a geometrically principled positional encoding for vision transformers with Weierstrass elliptic functions

  • 基于魏尔斯特拉斯椭圆函数,直接在复数域建模二维坐标关系。
  • 在CIFAR-100上达到63.78%准确率(ViT-Tiny)和93.28%(ViT-Base)。
  • 数学证明距离衰减特性,适合关注几何先验的视觉任务研究者。

视觉变压器在计算机视觉任务中表现卓越,但其依赖可学习的一维位置嵌入,通过补丁展平破坏了图像固有的二维空间结构。传统位置编码方法缺乏几何约束,无法建立欧氏空间距离与序列索引距离之间的单调对应,限制了模型对空间邻近先验的有效利用。本文提出魏尔斯特拉斯椭圆函数位置编码(WEF-PE),一种数学上严谨的方法,通过自然复数域表示直接处理二维坐标,椭圆函数的双重周期性与视觉数据中常见的平移不变模式高度契合。该方法利用椭圆函数的非线性几何特性自然编码空间距离关系,且其代数加法公式可直接从绝对编码推导任意补丁对的相对位置信息。大量实验表明,WEF-PE在多种场景下表现优越:在无预训练的ViT-Tiny架构上于CIFAR-100上达到63.78%准确率,在微调设置下的ViT-Base达到93.28%,并在VTAB-1k基准测试中持续提升。理论分析通过严格数学证明确认了距离衰减特性,注意力可视化显示其具备更强的几何归纳偏置和更连贯的语义聚焦,优于传统方法。代码已开源。

原文摘要 · Abstract (English)

Vision Transformers have demonstrated remarkable success in computer vision tasks, yet their reliance on learnable one-dimensional positional embeddings fundamentally disrupts the inherent two-dimensional spatial structure of images through patch flattening procedures. Traditional positional encoding approaches lack geometric constraints and fail to establish monotonic correspondence between Euclidean spatial distances and sequential index distances, thereby limiting the model's capacity to leverage spatial proximity priors effectively. We propose Weierstrass Elliptic Function Positional Encoding (WEF-PE), a mathematically principled approach that directly addresses two-dimensional coordinates through natural complex domain representation, where the doubly periodic properties of elliptic functions align remarkably with translational invariance patterns commonly observed in visual data. Our method exploits the non-linear geometric nature of elliptic functions to encode spatial distance relationships naturally, while the algebraic addition formula enables direct derivation of relative positional information between arbitrary patch pairs from their absolute encodings. Comprehensive experiments demonstrate that WEF-PE achieves superior performance across diverse scenarios, including 63.78\% accuracy on CIFAR-100 from-scratch training with ViT-Tiny architecture, 93.28\% on CIFAR-100 fine-tuning with ViT-Base, and consistent improvements on VTAB-1k benchmark tasks. Theoretical analysis confirms the distance-decay property through rigorous mathematical proof, while attention visualization reveals enhanced geometric inductive bias and more coherent semantic focus compared to conventional approaches.The source code implementing the methods described in this paper is publicly available on GitHub.

视觉Transformer位置编码椭圆函数几何先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。