用频谱字典取代卷积和自注意力，实现高效可解释的视觉语言模型。
From Pixels and Words to Waves: A Unified Framework for Spectral Dictionary vLLMs
- 用可学习的频率原子稀疏表示图像块和词元，替代传统卷积与自注意力。
- 1.1B参数模型在MS-COCO上达BLEU-4 39.2、CIDEr 127.5，VQAv2准确率50.3%。
- 比PaLI-3少60%参数、2.3倍低显存、2.2倍快推理，性能接近BLIP-2的85%。
视觉语言模型（VLMs）将计算机视觉与自然语言处理统一于单一架构中，以解析和描述图像。当前主流系统依赖两个计算开销大的组件：视觉编码器中的卷积与多模态融合中的二次自注意力。本文提出一种频谱字典标记混合器，将每个图像块或词元表示为可学习频率原子的稀疏组合。我们的1.1B参数原型模型SDict-VLM在MS-COCO图像描述任务上达到BLEU-4 39.2、CIDEr 127.5、SPICE 27.0，VQAv2准确率为50.3%。该性能接近BLIP-2的85%，但仅使用其60%参数，峰值显存减少2.3倍，推理速度提升2.2倍。据我们所知，这是首个同时摒弃卷积与自注意力且媲美中等规模Transformer基线的VLM。该方法具有O(L log L)复杂度，共享频率字典实现了清晰的跨模态对齐，并可在精度与计算量间灵活权衡，为高效且可解释的VLM铺平道路。
原文摘要 · Abstract (English)
Vision-language models (VLMs) unify computer vision and natural language processing in a single architecture capable of interpreting and describing images. Most state-of-the-art systems rely on two computationally intensive components: convolutions in the vision encoder and quadratic self-attention for multimodal fusion. This work removes both by introducing a spectral dictionary token mixer, which represents each image patch or wordpiece as a sparse combination of learnable frequency atoms. Our 1.1B-parameter prototype, SDict-VLM, achieves BLEU-4 of 39.2, CIDEr of 127.5, and SPICE of 27.0 on MS-COCO captioning, along with 50.3 percent accuracy on VQAv2. These results close approximately 85 percent of the performance gap to BLIP-2 while using 60 percent fewer parameters, 2.3 times less peak GPU memory, and 2.2 times faster inference than PaLI-3. To our knowledge, this is the first VLM to eliminate both convolutions and self-attention while matching mid-scale transformer baselines. In addition to its O(L log L) complexity, the shared frequency dictionary enables transparent cross-modal alignment and offers a tunable trade-off between accuracy and compute, paving the way for efficient and interpretable VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。