用几何三角面积约束提升多模态检索对齐效果
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
- 通过最小化模态三元组在超球面的面积实现整体对齐
- 零样本下最高提升8.7个Recall@1，但效果依赖领域
- 几何对齐有效但优化敏感，需配合领域微调
多模态对齐对弥合信息检索中的语义鸿沟至关重要。传统成对策略存在几何盲区：虽能对齐锚点模态（如文本）与其他模态，却缺乏对周边模态（如视频与音频）间相互一致性的约束。TRIANGLE框架通过最小化模态三元组在超球面上的面积来强制整体对齐。本可复现性研究验证了该几何目标在检索任务中的鲁棒性。结果表明，TRIANGLE在零样本设置下优于成对基线，Recall@1最高提升+8.7点，但收益具有领域依赖性。然而，我们未能复现报告的从头训练结果。使用合成数据集分析表明，这源于联合优化几何对齐与数据-文本匹配（DTM）损失时的不稳定性。此外，余弦正则化主要稳定文本到视频检索，而领域监督微调虽增强几何优势，却降低跨数据集泛化能力。研究支持几何对齐的有效性，同时揭示关键优化敏感性。代码见https://github.com/ARIJIT00171/RE-TRIANGLE。
原文摘要 · Abstract (English)
Multimodal alignment is critical for bridging the semantic gap in information retrieval. However, traditional pairwise strategies introduce a geometric blind spot: while they align anchor modalities (e.g., text) with others, they lack constraints to enforce mutual consistency between peripheral modalities (e.g., video and audio). The TRIANGLE framework addresses this by minimizing the area of modality triplets on a hypersphere to enforce holistic alignment. In this reproducibility study, we verify the robustness of this geometric objective for retrieval tasks. We confirm that TRIANGLE outperforms pairwise baselines in zero-shot settings, achieving Recall@1 gains of up to +8.7 points, though benefits are domain-dependent. However, we fail to reproduce the reported learning-from-scratch results. Analysis using a synthetic toy dataset attributes this to instability when jointly optimizing geometric alignment with Data-Text Matching (DTM) loss. Furthermore, we find that cosine regularization primarily stabilizes text-to-video retrieval, and fine-tuning with domain supervision amplifies geometric benefits but reduces cross-dataset generalization. Our findings support the efficacy of geometric alignment while highlighting critical optimization sensitivities. Code available at https://github.com/ARIJIT00171/RE-TRIANGLE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。