用微分注意力提升CLIP的图文理解能力，效果更好且不增加计算负担
DiffCLIP: Differential Attention Meets CLIP
- 将语言模型中的微分注意力引入CLIP双编码器架构
- 零样本分类、检索等任务上全面超越基线模型
- 仅增少量参数，计算开销几乎不变，适合高效多模态应用
我们提出DiffCLIP，一种将微分注意力机制引入CLIP架构的新型视觉-语言模型。微分注意力原用于大语言模型，可增强相关上下文并抑制噪声信息。本文将其集成至CLIP的图像与文本双编码器框架中。仅需少量额外参数，DiffCLIP在图像-文本理解任务上表现显著优于基准模型。在零样本分类、检索及鲁棒性测试中，性能持续领先。值得注意的是，这些提升几乎不增加计算开销，证明微分注意力能有效增强多模态表示而不牺牲效率。代码已开源：https://github.com/hammoudhasan/DiffCLIP。
原文摘要 · Abstract (English)
We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures. Differential attention was originally developed for large language models to amplify relevant context while canceling out noisy information. In this work, we integrate this mechanism into CLIP's dual encoder (image and text) framework. With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks. Across zero-shot classification, retrieval, and robustness benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably, these gains come with negligible computational overhead, demonstrating that differential attention can significantly enhance multi-modal representations without sacrificing efficiency. Code can be found at https://github.com/hammoudhasan/DiffCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。