arXiv:2602.23588cs.CVcs.AI2026-02

不微调模型也能实现图文对齐,用高维符号运算生成语义更准的图像描述。

Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

  • 将视觉与语言模型嵌入投影到高维空间,用符号操作构建跨模态关联
  • 性能接近端到端训练方法,且生成描述比零样本基线更语义化
  • 适合追求高效、低资源图文生成的开发者或研究者

视觉与语言的大规模单模态基础模型蕴含丰富语义结构,但传统对齐需计算密集的多模态微调,依赖大规模参数更新,资源消耗大且易破坏预训练表示。新证据表明,独立训练的基础模型可能已具备潜在语义兼容性,反映其建模数据的共享结构。这引出核心问题:能否不修改模型自身实现跨模态对齐?本文提出HDFLIM(HyperDimensional computing with Frozen Language and Image Models),在保持视觉与语言模型完全冻结的前提下,建立跨模态映射。HDFLIM将单模态嵌入投影至共享高维空间,利用轻量级符号操作——绑定、捆绑与基于相似性的检索,在单次数据遍历中构建关联的跨模态表示。图像描述生成基于高维记忆检索,而非迭代梯度优化。实验表明,HDFLIM性能可媲美端到端视觉-语言训练方法,生成描述更具语义根基,优于零样本基线。通过解耦对齐与参数微调,结果表明,基础模型间的语义映射可通过高维编码的符号操作实现。本工作为基础模型对齐提供新范式:以结构化表征映射替代大规模重训练。代码库见 https://github.com/Abhishek-Dalvi410/HDFLIM。

原文摘要 · Abstract (English)

Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.

图文对齐高维计算冻结模型生成描述

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。