解决图像检索中多模态对齐难题,提升跨模态匹配精度
CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval
- 用思维链引导生成语义一致的描述,建立模态对称性
- 双塔架构共享参数编码器,减少特征空间错位
- 动态记忆库提供高质量负样本,适配模型演化过程
组合图像检索(CIR)允许用户通过参考图像和操作文本搜索目标图像,相比单模态检索系统具有显著优势。然而,现有CIR方法存在表征空间碎片化问题:查询与目标包含异构模态,由不同编码器处理,导致模型仅能通过后期对齐来弥合错位表征空间,从根本上限制了检索性能。这种架构不对称在特征空间中表现为三个明显分离的聚类,直接证明异构模态从初始化即造成根本性表征错位。本文提出CSMCIR,一种统一表征框架,通过三个协同组件实现高效查询-目标对齐。首先,引入多层级思维链(MCoT)提示策略,引导多模态大语言模型生成判别性强、语义兼容的目标图像描述,建立模态对称性。在此基础上,设计对称双塔架构,查询与目标侧均使用相同共享参数的Q-Former进行跨模态编码,确保特征表示一致性,进一步缩小对齐差距。最终,该架构对称性支持基于熵的时序动态记忆库策略,持续提供高质量负样本,同时保持与模型状态演化的同步。在四个基准数据集上的大量实验表明,CSMCIR达到当前最优性能,且训练效率更优。全面消融实验证实了各组件的有效性。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text, offering substantial advantages over single-modality retrieval systems. However, existing CIR methods suffer from representation space fragmentation: queries and targets comprise heterogeneous modalities and are processed by distinct encoders, forcing models to bridge misaligned representation spaces only through post-hoc alignment, which fundamentally limits retrieval performance. This architectural asymmetry manifests as three distinct, well-separated clusters in the feature space, directly demonstrating how heterogeneous modalities create fundamentally misaligned representation spaces from initialization. In this work, we propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components. First, we introduce a Multi-level Chain-of-Thought (MCoT) prompting strategy that guides Multimodal Large Language Models to generate discriminative, semantically compatible captions for target images, establishing modal symmetry. Building upon this, we design a symmetric dual-tower architecture where both query and target sides utilize the identical shared-parameter Q-Former for cross-modal encoding, ensuring consistent feature representations and further reducing the alignment gap. Finally, this architectural symmetry enables an entropy-based, temporally dynamic Memory Bank strategy that provides high-quality negative samples while maintaining consistency with the evolving model state. Extensive experiments on four benchmark datasets demonstrate that our CSMCIR achieves state-of-the-art performance with superior training efficiency. Comprehensive ablation studies further validate the effectiveness of each proposed component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。