用变分自编码器生成动态视觉提示，提升模型对图像细节的适应能力
Visual Variational Autoencoder Prompt Tuning
- 通过变分自编码器学习图像特有特征，生成随输入变化的动态提示
- 在三个基准测试中平均提升2.0%，在HTA上比VPT-Deep高3.2%
- 适合需要高效微调且关注图像细节差异的应用场景
参数高效微调（PEFT）已成为适应大型视觉变换器到下游任务的关键方法，避免全量微调带来的高昂计算成本。现有视觉提示微调（VPT）方法多依赖静态、领域特定的提示，难以捕捉单个实例中的丰富视觉多样性。本文提出V²APT（视觉变分自编码器提示微调），一种新框架，利用变分自编码器架构生成动态、输入相关的提示。通过学习图像特有特征的潜在表示并解码为定制化提示，V²APT能够适应每个输入的独特视觉特性。在FGVC、HTA和VTAB-1k基准上的大量实验表明，该方法持续优于当前最先进的PEFT方法。值得注意的是，V²APT在HTA上相较VPT-Deep实现+3.2%的提升，三个数据集上平均性能提高+2.0%。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large vision transformers to downstream tasks without the prohibitive computational costs of full fine-tuning. While existing visual prompt tuning (VPT) methods have made significant strides, they predominantly rely on static, domain-specific prompts that fail to capture the rich visual diversity within individual instances. This paper introduces V$^2$APT (Visual Variational Autoencoder Prompt Tuning), a novel framework that generates dynamic, input-dependent prompts using a variational autoencoder architecture. By learning a latent representation of image-specific features and decoding them into customized prompts, V$^2$APT adapts to the unique visual characteristics of each input. Extensive experiments on FGVC, HTA, and VTAB-1k benchmarks demonstrate that our approach consistently outperforms state-of-the-art PEFT methods. Notably, V$^2$APT achieves +3.2\% improvement over VPT-Deep on HTA, with an average performance gain of +2.0\% across all three datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。