用最优传输约束防止视觉语言模型适配时遗忘预训练知识
Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
- 引入最优传输机制约束视觉与文本特征分布一致性
- 在多个数据集上实现更优的零样本泛化与跨域适应能力
- 无需额外增强或集成，适合高效微调场景
视觉语言模型（如CLIP）在下游任务中表现强劲，但微调时易出现过拟合且损害零样本泛化能力。提示学习作为高效适配策略，仍存在知识遗忘问题。本文提出一种基于最优传输（OT）的提示学习框架，通过保持预训练与微调后模型特征分布的结构一致性，缓解遗忘。相比传统逐点约束，OT能捕捉实例间关系，拓展提示调优的可行参数空间，实现适应性与泛化性的更好平衡。方法同时对视觉与文本表示施加联合约束，确保特征整体对齐。在多个基准数据集上的实验证明，该方法在基础到新类别泛化、跨数据集评估和领域泛化任务中均优于现有提示学习方法，且无需额外数据增强或集成技术。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks. Prompt learning has emerged as an efficient and effective strategy to adapt VLMs while preserving their pre-trained knowledge. However, existing methods still lead to overfitting and degrade zero-shot generalization. To address this challenge, we propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistency of feature distributions between pre-trained and fine-tuned models. Unlike conventional point-wise constraints, OT naturally captures cross-instance relationships and expands the feasible parameter space for prompt tuning, allowing a better trade-off between adaptation and generalization. Our approach enforces joint constraints on both vision and text representations, ensuring a holistic feature alignment. Extensive experiments on benchmark datasets demonstrate that our simple yet effective method can outperform existing prompt learning strategies in base-to-novel generalization, cross-dataset evaluation, and domain generalization without additional augmentation or ensemble techniques. The code is available at https://github.com/ChongQingNoSubway/Prompt-OT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。