用超边结构统一优化视觉变压器的特征,提升细调效率与一致性。
Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

- 在超边空间而非单个令牌间进行参数高效微调,引入结构化归纳偏置。
- 在多个视觉基准上优于主流基线,尤其在需要结构推理的任务中提升显著。
- 适合关注视觉模型细调效率与结构一致性的研究者或工程应用。
参数高效微调(PEFT)已成为将大型预训练视觉变压器(ViTs)适配到下游任务的实用方案,仅更新少量参数。然而,现有基于适配器的方法对每个令牌独立进行适应,隐含假设令牌修正应孤立学习。这种逐令牌设定忽略了视觉场景中自然存在的令牌间结构关系,可能导致冗余更新和空间不一致的特征优化。本文重新审视参数高效适配器的设计,提出在超边空间而非令牌空间进行适应。我们提出HyperAdapter,一种基于超图的适配器架构,通过软令牌路由实现结构化、群体感知的适应。HyperAdapter使用基于原型的分配在ViT令牌上构建软超图,将令牌特征聚合为潜在超边表示,在超边层面应用轻量级瓶颈适配,并通过超图关联结构将更新扩散回令牌。该设计在保持标准适配器模块化与高效性的同时,显式注入结构归纳偏置。在多种视觉基准上的大量实验表明,结构化超边适应在同等参数预算下持续优于强基线,尤其在需要结构推理的任务中表现突出。结果表明,适配空间的选择是视觉变压器参数高效迁移中一个关键但被忽视的维度。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has become a practical solution for adapting large pretrained vision transformers (ViTs) to downstream tasks while updating only a small subset of parameters. However, existing adapter-based methods perform adaptation independently for each token, implicitly assuming that token refinements should be learned in isolation. This token-wise formulation overlooks the structured relationships among tokens that naturally arise in visual scenes, potentially leading to redundant updates and spatially inconsistent feature refinement. In this work, we revisit the design of parameter-efficient adapters and propose to perform adaptation in hyperedge space rather than token space. We introduce HyperAdapter, a hypergraph-based adapter architecture that enables structured, group-aware adaptation through soft token routing. HyperAdapter constructs a soft hypergraph over ViT tokens using prototype-based assignments, aggregates token features into latent hyperedge representations, applies lightweight bottleneck adaptation at the hyperedge level, and diffuses the resulting updates back to tokens via the hypergraph incidence structure. This design injects an explicit structural inductive bias into PEFT while preserving the modularity and efficiency of standard adapters. Extensive experiments across diverse visual benchmarks demonstrate that structured hyperedge adaptation consistently outperforms strong PEFT baselines under comparable parameter budgets, with particularly pronounced gains on tasks requiring structured reasoning. Our results suggest that the choice of adaptation space is a critical yet underexplored dimension in parameter-efficient transfer for ViTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。