通过幂集对齐提升视觉语言模型的组合理解能力
PowerCLIP: Powerset Alignment for Contrastive Pre-Training
- 用幂集构建图像区域与文本短语的全组合对齐
- 在zero-shot分类和检索任务上超越现有方法
- 引入非线性聚合器降低计算复杂度至线性
对比学习型视觉语言预训练框架（如CLIP）在多种视觉语言任务中展现出出色的零样本性能。近期研究发现，将文本词元与特定图像区域对齐有助于实现细粒度的组合理解。然而，如何捕捉跨越多个图像区域的组合语义仍具挑战。为此，我们提出PowerCLIP，一种基于幂集对齐的新型对比预训练框架，通过最小化图像区域幂集与文本解析树之间的损失，全面优化区域到短语的对齐。由于直接构造幂集会导致区域子集数量呈组合爆炸，计算成本呈指数级增长，我们引入高效的非线性聚合器（NLAs），将复杂度从O(2^M)降至O(M)，同时可任意逼近精确损失值。大量实验表明，PowerCLIP在零样本分类与检索任务中优于当前最优方法，验证了其组合性与鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Code is available at https://github.com/Masakichi210/PowerCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。