arXiv:2409.02838cs.CV2024-09被引 3

让适配器随输入动态变化,用更少参数实现更好性能。

iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation

  • 通过输入条件网络生成动态卷积核,实现逐样本特征变换。
  • 仅调1.6%~2.8%参数,在多个任务上达到甚至超过全微调效果。
  • 适合需要高效微调且对任务细节敏感的下游应用。

基于预训练编码器和特定任务解码器的迁移学习在深度模型不断增长时变得日益复杂。参数高效的微调(PEFT)方法通过使用小型可学习层组成的适配器,成为全微调(FFT)的替代方案,在保持高训练效率的同时实现了相当的性能。然而,传统适配器对输入实例缺乏灵活性,限制了其在多样化下游任务中学习任务特异性信息的能力。本文提出一种新型PEFT方法——iConFormer,采用输入条件的动态适配器。通过在动态适配器中引入输入条件网络(iCoN),实现实例级特征变换:iCoN为每个特征生成通道级卷积核,并通过自适应卷积过程进行转换,有效捕捉任务特异性和细粒度细节。实验表明,仅需微调1.6%至2.8%的Transformer主干参数,iConFormer在单目深度估计和语义分割任务上性能接近全微调,在图像分类和实例分割任务上则表现更优。同时,该方法在所有上述任务中均持续优于近期其他PEFT方法。

原文摘要 · Abstract (English)

Transfer learning based on full fine-tuning (FFT) of the pre-trained encoder and task-specific decoder becomes increasingly complex as deep models grow exponentially. Parameter efficient fine-tuning (PEFT) approaches using adapters consisting of small learnable layers have emerged as an alternative to FFT, achieving comparable performance while maintaining high training efficiency. However, the inflexibility of the adapter with respect to input instances limits its capability of learning task-specific information in diverse downstream tasks. In this paper, we propose a novel PEFT approach, input-Conditioned transFormer, termed iConFormer, that leverages a dynamic adapter conditioned on the input instances. To secure flexible learning ability on input instances in various downstream tasks, we introduce an input-Conditioned Network (iCoN) in the dynamic adapter that enables instance-level feature transformation. To be specific, iCoN generates channel-wise convolutional kernels for each feature and transform it using adaptive convolution process to effectively capture task-specific and fine-grained details tailor to downstream tasks. Experimental results demonstrate that by tuning just 1.6% to 2.8% of the Transformer backbone parameters, iConFormer achieves performance comparable to FFT in monocular depth estimation and semantic segmentation, while outperforming it in image classification and instance segmentation. Also, the proposed method consistently outperforms recent PEFT methods for all the tasks mentioned above.

参数高效动态适配Transformer微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。