通过解耦图构建与卷积,实现视觉GNN在FPGA上的实时加速。
GraphLeap: Decoupling Graph Construction and Convolution for Vision GNN Acceleration on FPGA

- 用前一层特征预构建下一层图,实现图构建与特征更新并行
- 在FPGA上实现95.7倍于CPU、8.5倍于GPU的加速
- 无需显式存储边特征,适合资源受限的边缘部署
视觉图神经网络(ViGs)将图像表示为补丁令牌的图,通过动态图卷积实现自适应特征驱动的邻域建模。传统方法中每层需先构建依赖当前特征的k近邻图,再进行消息传递,该过程耗时占图卷积总时间的50%至95%,且随补丁数N呈O(N²)增长,并产生层间依赖。本文提出GraphLeap,通过解耦图构建与特征更新,使第ℓ层的特征更新使用第ℓ-1层的特征构建的图,同时利用第ℓ层特征构建第ℓ+1层的图,实现一跳前瞻图构建。此设计支持图构建与消息传递并发执行。虽使用历史特征可能带来轻微精度损失,但经数个周期轻量微调即可恢复原精度。基于GraphLeap,我们构建了首个面向视觉GNN的端到端FPGA加速器,采用流水线化、分层流水设计,融合节点级与通道级并行性,避免显式存储边特征,实现高效片上数据流。在Alveo U280 FPGA上对各向同性和金字塔型ViG模型的评估表明,GraphLeap相比CPU基线最高提速95.7倍,相比GPU基线最高提速8.5倍,验证了实时视觉GNN推理的可行性。
原文摘要 · Abstract (English)
Vision Graph Neural Networks (ViGs) represent an image as a graph of patch tokens, enabling adaptive, feature-driven neighborhoods. Unlike CNNs with fixed grid biases or Vision Transformers with global token interactions, ViGs rely on dynamic graph convolution: at each layer, a feature-dependent graph is built via k-nearest-neighbor (kNN) search on current patch features, followed by message passing. This per-layer graph construction is the main bottleneck, consuming 50--95\% of graph convolution time on CPUs and GPUs, scaling as $O(N^2)$ with the number of patches $N$, and creating a sequential dependency between graph construction and feature updates. We introduce GraphLeap, a simple reformulation that removes this dependency by decoupling graph construction from feature update across layers. GraphLeap performs the feature update at layer $\ell$ using a graph built from the previous layer's features, while simultaneously using the current layer's features to construct the graph for layer $\ell+1$. This one-layer-lookahead graph construction enables concurrent graph construction and message passing. Although using prior-layer features can introduce minor accuracy degradation, lightweight fine-tuning for a few epochs is sufficient to recover the original accuracy. Building on GraphLeap, we present the first end-to-end FPGA accelerator for Vision GNNs. Our streaming, layer-pipelined design overlaps a kNN graph construction engine with a feature update engine, exploits node- and channel-level parallelism, and enables efficient on-chip dataflow without explicit edge-feature materialization. Evaluated on isotropic and pyramidal ViG models on an Alveo U280 FPGA, GraphLeap achieves up to $95.7\times$ speedup over CPU and $8.5\times$ speedup over GPU baselines, demonstrating the feasibility of real-time Vision GNN inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。