让预训练的CLIP模型低成本进入双曲空间，提升零样本视觉问答性能。
HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
- 通过轻量微调将CLIP迁移到双曲空间，无需从头训练。
- 在多个零样本视觉问答任务上，最优模型比CLIP-B高1.9分。
- 适合需要高效适配新任务的视觉语言模型研究者。
近期表示学习进展表明，双曲几何可为CLIP模型中的欧氏嵌入提供更富表现力的替代方案，能更好捕捉层次结构并生成更有序的表示。然而，现有双曲CLIP变体需完全从头训练，计算成本高昂。本文提出HAC（Hyperbolic Adaptation of CLIP），一种参数高效的框架，使预训练的CLIP模型可通过轻量级微调进入双曲空间。我们将HAC应用于视觉问答（VQA），该任务要求模型理解视觉内容并与文本问题对齐。值得注意的是，HAC在与任何VQA基准无重叠的数据集上训练，实现严格的零样本评估，凸显其任务无关适应能力。我们在涵盖通用、推理和OCR类别的多样化VQA基准上评估HAC。HAC-S（小）和HAC-B（中）均持续超越欧氏基线及先前双曲方法，其中HAC-B在推理密集型任务上相比CLIP-B平均提升最高达+1.9点。代码已开源：https://github.com/fdibiton/HAC。
原文摘要 · Abstract (English)
Recent advances in representation learning have shown that hyperbolic geometry can offer a more expressive alternative to the Euclidean embeddings used in CLIP models, capturing hierarchical structures and leading to better-organized representations. However, current hyperbolic CLIP variants are trained entirely from scratch, which is computationally expensive and resource-intensive. In this work, we propose HAC (Hyperbolic Adaptation of CLIP), a parameter-efficient framework that enables pretrained CLIP models to transition into hyperbolic space via lightweight fine-tuning. We apply HAC to Visual Question Answering (VQA), where models must interpret visual elements and align them with textual queries. Notably, HAC's training is performed on a dataset with no overlap with any VQA benchmark, resulting in a strict zero-shot evaluation paradigm that underscores HAC's task-agnostic adaptability. We evaluate HAC across a diverse suite of VQA benchmarks spanning General, Reasoning, and OCR categories. Both HAC-S (small) and HAC-B (medium) consistently surpass Euclidean baselines and prior hyperbolic approaches, with HAC-B delivering up to a +1.9 point average improvement over CLIP-B on reasoning-intensive tasks. Our code is available at https://github.com/fdibiton/HAC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。