用张量网络显式建模语法结构，提升视觉语言模型对语序和动词语义的敏感度。
DisCoCLIP: A Distributional Compositional Tensor Network Encoder for Vision-Language Understanding
- 通过组合范畴语法解析句子，生成体现句法结构的分布式词张量。
- 在SVO-Probes上将动词准确率从77.6%提升至82.4%，SVO-Swap达93.7%。
- 参数量降至百万级，兼具高效性与可解释性，适合需要细粒度语义理解的任务。
当前视觉语言模型在大规模图像-文本对齐上表现优异，但常忽略语言的组合结构，导致在依赖词序和谓词-论元结构的任务中失效。我们提出DisCoCLIP，一种结合冻结的CLIP视觉变换器与新型张量网络文本编码器的多模态编码器，显式编码语法结构。句子经组合范畴语法解析后生成分布式词张量，其张量收缩过程对应句子的句法推导。为保持效率，高阶张量采用张量分解进行压缩，参数量从数千万降至不足一百万。通过自监督对比损失端到端训练，DisCoCLIP显著提升对动词语义和词序的敏感度：在SVO-Probes任务中动词准确率从77.6%提升至82.4%，ARO归因与关系得分分别提升超9%和4%，并在新提出的SVO-Swap基准上达到93.7%。结果表明，通过张量网络嵌入显式语言结构，可获得可解释、参数高效的表示，显著增强视觉语言任务中的组合推理能力。
原文摘要 · Abstract (English)
Recent vision-language models excel at large-scale image-text alignment but often neglect the compositional structure of language, leading to failures on tasks that hinge on word order and predicate-argument structure. We introduce DisCoCLIP, a multimodal encoder that combines a frozen CLIP vision transformer with a novel tensor network text encoder that explicitly encodes syntactic structure. Sentences are parsed with a Combinatory Categorial Grammar parser to yield distributional word tensors whose contractions mirror the sentence's grammatical derivation. To keep the model efficient, high-order tensors are factorized with tensor decompositions, reducing parameter count from tens of millions to under one million. Trained end-to-end with a self-supervised contrastive loss, DisCoCLIP markedly improves sensitivity to verb semantics and word order: it raises CLIP's SVO-Probes verb accuracy from 77.6% to 82.4%, boosts ARO attribution and relation scores by over 9% and 4%, and achieves 93.7% on a newly introduced SVO-Swap benchmark. These results demonstrate that embedding explicit linguistic structure via tensor networks yields interpretable, parameter-efficient representations that substantially improve compositional reasoning in vision-language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。