GSPN-2让视觉模型高效处理长序列,速度大幅提升且保持精度。
GSPN-2: Efficient Parallel Sequence Modeling
- 用单个2D核替代千次微启动,减少计算开销
- 共享内存缓存列激活,降低显存传输次数
- 紧凑通道传播设计,参数更少、适配注意力机制
高效视觉变压器在高分辨率图像和长视频应用中仍受限于计算瓶颈。广义空间传播网络(GSPN)通过线性扫描传播替代二次自注意力,使计算成本接近行列数的线性关系,同时保持精度。然而原GSPN实现仍存在三方面问题:(i) GPU核反复启动带来沉重开销,(ii) 全局显存数据传输过多,(iii) 每通道独立维护传播权重导致冗余计算。本文提出GSPN-2,进行算法与系统联合重构:将数千次微启动合并为单一2D核,为每通道切片显式绑定一个线程束,并在共享内存中暂存前一列激活值。模型层面引入紧凑通道传播策略,以统一结构替代各通道矩阵,减少参数量,并自然契合变压器注意力中的亲和图。实验表明,GSPN-2在图像分类与文生图任务中达到与变压器相当的精度,同时显著降低计算成本。GSPN-2通过结构化矩阵变换与GPU优化实现,为视觉建模全局空间上下文树立新效率标杆。
原文摘要 · Abstract (English)
Efficient vision transformer remains a bottleneck for high-resolution images and long-video related real-world applications. Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme, bringing the cost close to linear in the number of rows or columns, while retaining accuracy. Despite this advancement, the existing GSPN implementation still suffers from (i) heavy overhead due to repeatedly launching GPU kernels, (ii) excessive data transfers from global GPU memory, and (iii) redundant computations caused by maintaining separate propagation weights for each channel. We introduce GSPN-2, a joint algorithm-system redesign. In particular, we eliminate thousands of micro-launches from the previous implementation into one single 2D kernel, explicitly pin one warp to each channel slice, and stage the previous column's activations in shared memory. On the model side, we introduce a compact channel propagation strategy that replaces per-channel matrices, trimming parameters, and align naturally with the affinity map used in transformer attention. Experiments demonstrate GSPN-2's effectiveness across image classification and text-to-image synthesis tasks, matching transformer-level accuracy with significantly lower computational cost. GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications through its unique combination of structured matrix transformations and GPU-optimized implementation. Project page: https://whj363636.github.io/GSPN2/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。