为视觉图网络设计低秩语义提示，提升下游任务表现
Vision Graph Prompting via Semantic Low-Rank Decomposition
- 基于语义连通组件的低秩特性，分解并融合提示信息
- 在多个下游任务上逼近全量微调效果，保持参数高效
- 适合需要高效适配图结构视觉模型的研究者
视觉图神经网络（ViG）通过将图像表示为图结构，更自然地捕捉非规则语义模式，超越传统网格或序列表示。为高效适配下游任务，参数高效的微调技术如视觉提示日益重要。然而，现有提示方法主要针对Transformer架构，忽视了图结构中节点与边间的丰富拓扑关系，限制了复杂语义建模能力。本文提出专为视觉图结构设计的视觉图提示（VGP）框架。核心洞察是：图中语义相连的组件具有低秩特性。基于此，我们提出语义低秩提示方法，将低秩语义特征分解并融入图拓扑中的提示，同时捕获全局结构模式与细粒度语义依赖。大量实验表明，该方法显著提升ViG在多样化下游任务上的迁移性能，达到接近全量微调的效果，同时保持参数效率。代码已公开于https://github.com/zhoujiahuan1991/ICML2025-VGP。
原文摘要 · Abstract (English)
Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditional grid or sequence-based representations. To efficiently adapt ViG to downstream tasks, parameter-efficient fine-tuning techniques like visual prompting become increasingly essential. However, existing prompting methods are primarily designed for Transformer-based models, neglecting the rich topological relationships among nodes and edges in graph-based representations, limiting their capacity to model complex semantics. In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures. Our core insight reveals that semantically connected components in the graph exhibit low-rank properties. Building on this observation, we introduce a semantic low-rank prompting method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-grained semantic dependencies. Extensive experiments demonstrate our method significantly improves ViG's transfer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our code is available at https://github.com/zhoujiahuan1991/ICML2025-VGP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。