用图结构在训练时提升少样本学习的原型质量,推理零开销。
Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters
- 训练时构建图文异构图教师,通过跨模态图推理捕捉细粒度关系。
- 在1-16样本基准上持续刷新少样本学习性能新纪录。
- 适合追求高效少样本适配且不希望增加推理延迟的研究者。
基于适配器的CLIP微调(如Tip-Adapter)是高效的少样本学习方法,通过缓存支持集特征实现快速原型匹配。然而,这些方法依赖全局单模态特征向量,忽略了视觉补丁间的细粒度关系及其与类别文本的结构对齐。为弥补这一差距而不增加推理成本,我们提出一种新颖的非对称训练仅框架。不修改轻量级适配器,而是训练阶段引入一个高容量的异构图教师模型。该教师模型(i)将多尺度视觉补丁与文本提示整合为统一图结构,(ii)通过模态感知图变换器(MGT)进行深度跨模态推理,(iii)采用判别性节点过滤提取高保真类别特征。关键在于,我们采用缓存感知双目标策略,将这种关系知识直接监督注入到Tip-Adapter的键值缓存中,从而在不改变测试流程的前提下有效升级原型。图教师在测试时被丢弃,因此推理与原始Tip-Adapter完全一致,无额外延迟或内存开销。在标准1-16样本基准上,本方法持续达到新最佳性能。消融实验验证了辅助图监督、文本引导推理和节点过滤是鲁棒少样本适应的关键组件。代码已开源:https://github.com/MR-Sherif/TOGA.git。
原文摘要 · Abstract (English)
Recent adapter-based CLIP tuning (e.g., Tip-Adapter) is a strong few-shot learner, achieving efficiency by caching support features for fast prototype matching. However, these methods rely on global uni-modal feature vectors, overlooking fine-grained patch relations and their structural alignment with class text. To bridge this gap without incurring inference costs, we introduce a novel asymmetric training-only framework. Instead of altering the lightweight adapter, we construct a high-capacity auxiliary Heterogeneous Graph Teacher that operates solely during training. This teacher (i) integrates multi-scale visual patches and text prompts into a unified graph, (ii) performs deep cross-modal reasoning via a Modality-aware Graph Transformer (MGT), and (iii) applies discriminative node filtering to extract high-fidelity class features. Crucially, we employ a cache-aware dual-objective strategy to supervise this relational knowledge directly into the Tip-Adapter's key-value cache, effectively upgrading the prototypes while the graph teacher is discarded at test time. Thus, inference remains identical to Tip-Adapter with zero extra latency or memory. Across standard 1-16-shot benchmarks, our method consistently establishes a new state-of-the-art. Ablations confirm that the auxiliary graph supervision, text-guided reasoning, and node filtering are the essential ingredients for robust few-shot adaptation. Code is available at https://github.com/MR-Sherif/TOGA.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。