arXiv:2605.17727cs.CV2026-05中稿 · EMNLP

让视觉语言模型的嵌入长度可控制语义粒度,实现从粗到细的语义渐进解析。

GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations

论文配图:GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations
图 1 · 摘自论文原文
  • 通过共享正交前缀变换,将固定长度嵌入转为可截断的语义接口。
  • 在COCO/Flickr30K上实现53.01的阶梯得分与89.76的硬负样本选择性。
  • 保持原空间几何不变,支持零样本迁移与多粒度语义提取,适合需要可控语义解析的场景。

冻结的视觉语言嵌入包含多个语义层次的信号,从物体身份到属性、关系及完整描述语义,但这些信号通过固定长度向量接口暴露。本文研究是否可将嵌入长度转化为可控的语义访问接口。提出GraSP-VL,学习对冻结的VLM嵌入进行共享近正交前缀变换。GraSP-VL实现了「语义套娃」接口:短前缀对应粗粒度语义角色,长前缀逐步揭示更精细的语言-图像对齐差异。该变换共享于图像与文本嵌入,且保留全维几何结构,无需重写原VLM空间即可改变前缀行为。在20,147个样本的COCO/Flickr30K标注池上,GraSP-VL达到53.01的阶梯得分和89.76的硬负样本选择性,同时保持全空间漂移低于$10^{-6}$。还成功迁移到SugarCrepe-clean数据集,实现86.03的物体识别准确率与11.96的平均外部涌现,且维持全维零样本CIFAR-100准确率。结果表明,冻结的VLM嵌入可被重构为可截断的语义前缀接口,而不仅限于压缩。

原文摘要 · Abstract (English)

Frozen vision-language embeddings contain signals at multiple semantic resolutions, from object identity to attributes, relations, and full-caption meaning, but they expose these signals through a fixed-length vector interface. We study whether embedding length can be turned into a controllable semantic access interface. We propose \textbf{GraSP-VL}, which learns a shared near-orthogonal prefix transform over frozen VLM embeddings. GraSP-VL instantiates a \textbf{Semantic Matryoshka} interface: short prefixes are assigned coarse semantic roles, while longer prefixes progressively expose finer language-grounded distinctions. Because the transform is shared across image and text embeddings and preserves full-dimensional geometry, prefix behavior changes without rewriting the original VLM space. On a 20,147-example COCO/Flickr30K annotation pool, GraSP-VL reaches a staircase score of 53.01 and hard-negative selectivity of 89.76, while keeping full-space drift below $10^{-6}$. It also transfers to SugarCrepe-clean with 86.03 object accuracy and 11.96 mean external emergence, and preserves full-dimensional zero-shot CIFAR-100 accuracy. These results show that frozen VLM embeddings can be reorganized into a truncatable semantic prefix interface rather than merely compressed.

视觉语言嵌入接口语义粒度前缀

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。