arXiv:2505.14062cs.CV2025-05被引 1

用希尔伯特分形曲线重排图像块,让视觉Mamba在高分辨率下仍保持高效准确。

FractalMamba++: Scaling Vision Mamba Across Resolutions via Hilbert Fractal Geometry

  • 用希尔伯特分形曲线重排图像块序列,更好保留二维空间邻近性。
  • 在多个数据集上,高分辨率下相比同类模型提升1.8%~3.2%准确率。
  • 适合需要处理高清图像的视觉任务,如遥感和医学影像分析。

视觉Mamba对长视觉序列具有线性复杂度,但其性能高度依赖于将二维图像块网格转化为一维状态空间递推的方式。栅格扫描破坏了空间连续性,当推理分辨率超过训练网格时,二维局部性与一维状态传播的不匹配愈发严重。本文提出FractalMamba++,一个基于希尔伯特分形几何统一原则的可扩展视觉骨干网络:希尔伯特曲线的递归自相似结构决定了图像块的序列化方式、长程状态跳跃的插入位置以及位置关系的编码方式。首先,基于希尔伯特曲线的分形序列化比线性扫描更忠实保留2D局部邻域,并在不同分辨率下保持一致的邻域统计特性。其次,分形层级跳接(FHSC)从希尔伯特递归层级中提取一组确定性的状态注入路径,无需运行时搜索或专用CUDA内核即可缓解长序列信息衰减。第三,分形感知2D旋转位置编码(FA-RoPE)结合归一化2D坐标与分形层级,使特征交互基于实际空间邻近性和递归结构角色,而非序列化后的1D距离。在ImageNet-1K分类、COCO检测与实例分割、ADE20K语义分割及LEVIR-CD+遥感变化检测上的大量实验表明,FractalMamba++在现有Mamba基视觉骨干中表现更优,尤其在高分辨率输入下优势显著。

原文摘要 · Abstract (English)

Vision Mamba offers linear complexity for long visual sequences, yet its performance depends critically on how a two-dimensional patch grid is serialized into a one-dimensional state-space recurrence. Raster-style scans disrupt spatial continuity, and the mismatch between 2D locality and 1D state propagation becomes increasingly severe when the inference resolution grows beyond the training grid. This paper presents FractalMamba++, a resolution-scalable vision backbone organized around a single geometric principle: the recursive self-similar structure of the Hilbert curve determines how patches are serialized, where long-range state shortcuts are inserted, and how positional relations are encoded. First, Hilbert-curve-based Fractal Serialization preserves local 2D neighborhoods more faithfully than linear scans and provides consistent neighborhood statistics across resolutions. Second, the Fractal Hierarchy Skip Connection (FHSC) derives a compact set of deterministic state-injection routes from Hilbert recursion levels, mitigating long-sequence information fading without runtime search, hand-derived gradients, or dedicated CUDA kernels. Third, Fractal-Aware 2D Rotary Position Encoding (FA-RoPE) combines normalized 2D coordinates with a fractal hierarchy level so that feature interactions depend on actual spatial proximity and recursive structural role rather than serialized 1D distance. Extensive experiments on ImageNet-1K classification, COCO detection and instance segmentation, ADE20K semantic segmentation, and LEVIR-CD+ remote sensing change detection show that FractalMamba++ improves over existing Mamba-based vision backbones, especially under high-resolution inputs.

视觉Mamba分形几何高分辨率位置编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。