用二维传播机制突破视觉大模型算力瓶颈,实现高效高分辨率建模。
Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

- 基于2D网格的线扫描递归传播,避免序列化损失空间结构。
- 提升至40-52倍速度,2K分辨率下推理快4倍且无需分块。
- 可替代注意力结构,适合大规模视觉预训练与高分辨率迁移。
视觉基础模型受限于自注意力的二次方计算开销,制约了可用分辨率并增加了大规模预训练成本。虽然线性注意力和状态空间模型能降低开销,但通常将图像序列化为一维标记流,削弱了对视觉任务至关重要的二维空间结构。广义空间传播网络(GSPN)通过在二维网格上直接传播上下文,实现近线性复杂度且无需位置嵌入,但尚未广泛用于基础规模编码器。本文提出C-GSPN,一种基于二维空间传播的基础规模视觉编码器。通过三项改进使该算子实用:(1) 快速的GSPN CUDA内核,融合每步调用为单一波前专用实现,采用共享内存分块、合并访问及紧凑多通道传播,达到峰值内存带宽90%以上,运行速度比原始GSPN快40–52倍;(2) 压缩潜空间传播模块,融合归一化操作,将内核级加速转化为模块与模型级效率;(3) 两阶段跨算子蒸馏方案,从注意力教师模型中训练新架构,无需从零开始的大规模训练。使用6亿张图像-文本对蒸馏后,C-GSPN以15%更少参数匹配同构的ViT基线,在ADE20K分割上提升+2.1%,高分辨率迁移仅需极少数据,2K分辨率下实现单次通过、无需分块的4倍端到端块加速。
原文摘要 · Abstract (English)
Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear attention and state-space models reduce this cost, but often serialize images into 1D token streams and weaken the 2D spatial structure important for vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional embeddings, but have seen little use as foundation-scale encoders. We present C-GSPN, a foundation-scale vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak memory bandwidth and running up to 40--52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator distillation recipe that trains the new architecture from an attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。