不改模型结构，用数据增强和正则化提升视觉语言模型提示调优效率。
MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
- 通过数据驱动增强优化初始数据分布，无需修改原模型架构。
- 引入可调节正则化模块，显著提升任务特征处理能力，训练成本低。
- 适合追求高效微调且不想改动模型结构的研究者或工程师。
尽管基于CLIP的提示调优能显著提升预训练视觉语言模型性能，现有研究多聚焦于重构模型架构，如增加损失计算或元网络，但这类方法通常导致复杂度上升和训练成本增加。为保持调优过程的高效性，我们提出即插即用的模型无关优化（MAO）用于提示调优。无需修改提示调优主干的任何组件，我们引入数据驱动增强框架以优化初始数据分布，并结合可调节正则化模块，增强任务特定特征处理流程，从而在保持低计算开销的同时提升整体性能。大量实验表明，MAO展现出卓越的性能与效率。代码已开源：https://github.com/JREion/M.A.O。
原文摘要 · Abstract (English)
Though CLIP-based prompt tuning significantly enhances pre-trained Vision-Language Models, existing research focuses on reconstructing the model architecture, e.g., additional loss calculation and meta-networks. These approaches generally lead to increased complexity and extended training cost. To maintain the efficiency of the tuning process, we propose plug-and-play Model-Agnostic Optimization (MAO) for prompt tuning. Without altering any components of the prompt tuning backbone, we introduce a Data-Driven Enhancement framework to optimize the distribution of the initial data, and incorporate an Alterable Regularization module to boost the task-specific feature processing pipeline, thereby improving overall performance while maintaining low computational cost. Extensive experiments on MAO demonstrate its outstanding performance and efficiency. The code of MAO is available at: https://github.com/JREion/M.A.O .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。