arXiv:2607.12418cs.CV2026-07

用量子计算提升视觉语言模型细粒度识别能力,解决相似类别区分难题。

MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning

论文配图:MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning
图 1 · 摘自论文原文
  • 先找最相关的Top-K类别作语义锚点,再用量子机制融合图文特征
  • 在15个数据集上显著提升细粒度分类准确率,参数量少于传统方法
  • 适合需要高精度小样本分类的视觉语言模型优化场景

大规模视觉语言模型在众多任务中展现出强大的迁移学习能力。针对少样本分类,我们发现这些模型能有效筛选候选类别,实现较高的Top-K准确率,但在视觉相似类别间的精细区分上表现不佳,导致Top-1性能不理想,如图1所示。现有VLM适配器研究多关注特征空间中视觉与文本表示的全局对齐,却未能利用语义相近类别来优化细粒度视觉表示。基于此,我们提出一种新的粗到精微调方法——多模态量子适配器(MQAdapter),引入量子计算以提升性能。具体而言,MQAdapter首先检索与输入图像最相似的Top-K类别作为语义锚点,再通过跨模态量子学习机制,在锚点引导下优化视觉特征。其核心是将视觉与文本特征编码为量子态,利用高维希尔伯特空间中的量子纠缠与叠加效应,有效建模更高阶的跨模态交互,生成比传统欧氏适配器更具判别性的表征。该方法参数高效,可与多种现有微调算法结合,进一步提升性能。在15个数据集上的评估验证了其有效性,同时所需可训练参数更少。

原文摘要 · Abstract (English)

Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate categories and thus achieve high Top-K accuracy. However, they often struggle with fine-grained discrimination among visually similar categories, resulting in unsatisfactory Top-1 performance, as shown in Figure 1. Existing studies on VLM adapters generally focus on global alignment between visual and textual representations in the feature space, but fail to exploit semantically similar categories to refine fine-grained visual representations. Based on these observations, we propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation, termed the Multi-Modal Quantum Adapter (MQAdapter). Specifically, MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors. It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors. The core of this mechanism is the encoding of visual and textual features into quantum states. By leveraging quantum entanglement and superposition in a high-dimensional Hilbert space, MQAdapter effectively models higher-order cross-modal interactions, producing more discriminative representations than traditional Euclidean adapters. MQAdapter is parameter-efficient and can be integrated with various existing fine-tuning algorithms to achieve further performance gains. Evaluations on 15 datasets demonstrate the effectiveness of MQAdapter while requiring fewer trainable parameters.

视觉语言模型量子计算少样本学习细粒度分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。