arXiv:2603.11542cs.CVcs.AI2026-03

ReHARK通过多阶段优化,让视觉语言模型在极少量数据下仍能稳定准确地适应新任务。

ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

  • 用混合先验与多尺度核函数构建全局正则化机制,解决边界偏差问题
  • 在11个基准上达到65.83%平均准确率,刷新单样本适配性能上限
  • 适合资源受限场景下的快速模型部署,尤其适合少样本应用

将大规模视觉-语言模型(如CLIP)适配到下游任务时,极低数据量(单样本)条件下的“稳定性-可塑性”困境始终存在。现有无训练方法(如Tip-Adapter)通常采用局部Nadaraya-Watson估计器,存在固有边界偏差且缺乏全局结构正则化。本文提出ReHARK(精炼混合自适应RBF核),通过再生核希尔伯特空间(RKHS)中的全局近端正则化重构少样本适配。引入多阶段精炼流程:(1)混合先验构建,融合零样本文本知识(来自CLIP和GPT-3)与视觉类别原型形成稳健语义-视觉锚点;(2)支持集增强(桥接),生成中间样本以平滑视觉与文本模态间过渡;(3)自适应分布校正,对齐测试特征统计与增强支持集以缓解域偏移;(4)多尺度RBF核,采用核函数集成捕捉跨尺度复杂特征几何结构。在11个多样化基准上进行广泛实验,验证其卓越的稳定性与准确性。ReHARK实现新的单样本适配最优性能,平均准确率达65.83%,显著优于现有基线。代码已公开于https://github.com/Jahid12012021/ReHARK。

原文摘要 · Abstract (English)

The adaptation of large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks with extremely limited data -- specifically in the one-shot regime -- is often hindered by a significant "Stability-Plasticity" dilemma. While efficient caching mechanisms have been introduced by training-free methods such as Tip-Adapter, these approaches often function as local Nadaraya-Watson estimators. Such estimators are characterized by inherent boundary bias and a lack of global structural regularization. In this paper, ReHARK (Refined Hybrid Adaptive RBF Kernels) is proposed as a synergistic training-free framework that reinterprets few-shot adaptation through global proximal regularization in a Reproducing Kernel Hilbert Space (RKHS). A multistage refinement pipeline is introduced, consisting of: (1) Hybrid Prior Construction, where zero-shot textual knowledge from CLIP and GPT-3 is fused with visual class prototypes to form a robust semantic-visual anchor; (2) Support Set Augmentation (Bridging), where intermediate samples are generated to smooth the transition between visual and textual modalities; (3) Adaptive Distribution Rectification, where test feature statistics are aligned with the augmented support set to mitigate domain shifts; and (4) Multi-Scale RBF Kernels, where an ensemble of kernels is employed to capture complex feature geometries across diverse scales. Superior stability and accuracy are demonstrated through extensive experiments on 11 diverse benchmarks. A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%, significantly outperforming existing baselines. Code is available at https://github.com/Jahid12012021/ReHARK.

少样本学习视觉语言模型无训练适配RBF核

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。