用交替训练提升提示词泛化能力,效果优于传统标签平滑。
Alternating Training-based Label Smoothing Enhances Prompt Generalization
- 交替使用硬标签和软标签训练提示词,缓解过自信问题。
- 结合类别/实例级软标签,使提示词在多个数据集上性能提升1.5%-3.2%。
- 适配主流提示调优方法,无需修改模型结构。
预训练视觉-语言模型展现出强大的零样本泛化能力。为进一步提升其对下游任务的适应性,提示调优作为一种参数高效微调方法应运而生。然而,提示词的泛化能力仍受限。标签平滑(LS)作为有效正则化技术,可防止模型过自信并提升泛化能力。本文发现,原始标签平滑反而削弱了提示调优的泛化性能。为此,提出交替训练标签平滑(ATLaS),通过交替使用标准一热标签和由标签平滑生成的软标签来监督提示调优。同时引入两类高效离线软标签:类别级软标签(CSL)与实例级软标签(ISL),分别提供类间或实例-类间关系信息。理论分析验证了方法性质。大量实验表明,结合CSL与ISL的ATLaS方法能持续提升提示调优的泛化性能,在多个基准测试中平均提升1.5%-3.2%。且该方法与主流提示调优方法高度兼容,可无缝集成。
原文摘要 · Abstract (English)
Recent advances in pre-trained vision-language models have demonstrated remarkable zero-shot generalization capabilities. To further enhance these models' adaptability to various downstream tasks, prompt tuning has emerged as a parameter-efficient fine-tuning method. However, despite its efficiency, the generalization ability of prompt remains limited. In contrast, label smoothing (LS) has been widely recognized as an effective regularization technique that prevents models from becoming over-confident and improves their generalization. This inspires us to explore the integration of LS with prompt tuning. However, we have observed that the vanilla LS even weakens the generalization ability of prompt tuning. To address this issue, we propose the Alternating Training-based Label Smoothing (ATLaS) method, which alternately trains with standard one-hot labels and soft labels generated by LS to supervise the prompt tuning. Moreover, we introduce two types of efficient offline soft labels, including Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL), to provide inter-class or instance-class relationships for prompt tuning. The theoretical properties of the proposed ATLaS method are analyzed. Extensive experiments demonstrate that the proposed ATLaS method, combined with CSL and ISL, consistently enhances the generalization performance of prompt tuning. Moreover, the proposed ATLaS method exhibits high compatibility with prevalent prompt tuning methods, enabling seamless integration into existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。