让视觉语言模型通信更高效,用稀疏编码压缩传输数据128倍。
Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents
- 对冻结的视觉语言模型激活值做后置稀疏自编码,实现高压缩比传输。
- 每标记仅保留4个活跃系数,传输字节数减少128倍,准确率下降不足0.1%。
- 仅用50个词典向量即可表达任务特征,适合低延迟、高带宽场景应用。
隐空间通信使异构视觉语言模型代理能够交换连续表示,而无需将视觉和推理状态序列化为文本。Vision Wormhole通过将视觉特征转换为通用潜在表示来实现此方法,但每个消息均以相同大小的稠密张量传输,无论内容如何。固定容量的稠密张量并不具备固定的有效信息密度:某些消息可能仅使用可用表征自由度的一小部分。这一观察表明通信通道具有显著可压缩性。我们通过对冻结的Vision Wormhole激活值拟合后置稀疏自编码器,评估了重建质量、下游任务性能、特征复用及标记级干预在九个推理基准上的表现。相较于原始float32传输,采用uint16索引/float16值的稀疏负载(每标记k=4个活跃系数)将传输字节数减少128倍。单次评估中,七项任务非AIME平均准确率从49.85%降至49.77%。拟合的4096维词典仅使用50个特征,任务级活跃集合的平均成对Jaccard相似度达0.906。这些测量结果证明了相对于原始传输方式的强大后置可压缩性,但尚未区分稀疏编码与位置选择、精度降低、低秩结构或SAE优化效果的增量贡献。结果推动开展匹配负载对比与动态适应信息使用的通信机制研究。
原文摘要 · Abstract (English)
Latent-space communication allows heterogeneous vision-language model agents to exchange continuous representations without serializing visual and reasoning states into text. Vision Wormhole realizes this approach by translating visual features into a universal latent representation that can be consumed by another model, but every message is transported as a dense tensor of the same size regardless of its content. A fixed-capacity dense tensor therefore need not have a fixed effective information density: some messages may use only a small fraction of the available representational degrees of freedom. This observation suggests that the communication channel may be substantially compressible. We study its redundancy by fitting a post-hoc sparse autoencoder to frozen Vision Wormhole activations and measuring reconstruction, downstream utility, feature reuse, and token-level interventions across nine reasoning benchmarks. Relative to the original float32 transport, a uint16-index/float16-value sparse payload with k=4 active coefficients per token reduces the transmitted bytes by 128x. In a single-run evaluation, the seven-task non-AIME mean accuracy changes from 49.85% to 49.77%. The fitted 4096-element dictionary uses only 50 features, and task-level active sets have a mean pairwise Jaccard similarity of 0.906. These measurements establish strong post-hoc compressibility relative to the original transport, but do not yet isolate the incremental contribution of sparse coding from position selection, reduced precision, low-rank structure, or SAE optimization effects. The results motivate matched-payload comparisons and communication mechanisms whose payload adapts to the information used by each message.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。