通过正交优化文本编码,提升视觉语言模型的分类性能。
ORION: ORthonormal Text Encoding for Universal VLM AdaptatION
- 用低秩适配优化文本嵌入,强制类别间正交
- 在11个基准上显著提升零样本、少样本性能
- 无需修改模型,可即插即用,适合迁移学习场景
视觉语言模型(VLMs)在多种任务中表现出色,但其性能受限于文本原型的质量与几何结构。标准零样本分类器依赖冻结的文本编码器和手工提示,可能产生相关或分离度弱的嵌入,影响任务判别能力。我们提出ORION,一种仅使用类别名称进行文本编码器微调的框架。该方法通过低秩适配优化一个新损失函数,包含两项:一项促进特定任务中各类别文本表示的成对正交性,另一项惩罚偏离初始类别原型的偏差。此外,我们给出了正交性惩罚的概率解释,通过胡亚格斯定理将其与最大似然估计(MLE)原则联系起来。我们在11个基准和三个大型VLM主干网络上进行了广泛实验,结果表明优化后的文本嵌入可有效替代标准CLIP原型。作为即插即用模块,应用于多种先进方法及不同预测设置(零样本、少样本、测试时自适应),ORION始终一致且显著提升性能。
原文摘要 · Abstract (English)
Vision language models (VLMs) have demonstrated remarkable generalization across diverse tasks, yet their performance remains constrained by the quality and geometry of the textual prototypes used to represent classes. Standard zero shot classifiers, derived from frozen text encoders and handcrafted prompts, may yield correlated or weakly separated embeddings that limit task specific discriminability. We introduce ORION, a text encoder fine tuning framework that improves pretrained VLMs using only class names. Our method optimizes, via low rank adaptation, a novel loss integrating two terms, one promoting pairwise orthogonality between the textual representations of the classes of a given task and the other penalizing deviations from the initial class prototypes. Furthermore, we provide a probabilistic interpretation of our orthogonality penalty, connecting it to the general maximum likelihood estimation (MLE) principle via Huygens theorem. We report extensive experiments on 11 benchmarks and three large VLM backbones, showing that the refined textual embeddings yield powerful replacements for the standard CLIP prototypes. Added as plug and play module on top of various state of the art methods, and across different prediction settings (zero shot, few shot and test time adaptation), ORION improves the performance consistently and significantly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。