arXiv:2608.10198cs.AI2026-08

让视觉语言模型通信更高效,用稀疏编码压缩传输数据128倍。

Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents

  • 对冻结的视觉语言模型激活值做后置稀疏自编码,实现高压缩比传输。
  • 每标记仅保留4个活跃系数,传输字节数减少128倍,准确率下降不足0.1%。
  • 仅用50个词典向量即可表达任务特征,适合低延迟、高带宽场景应用。

隐空间通信使异构视觉语言模型代理能够交换连续表示,而无需将视觉和推理状态序列化为文本。Vision Wormhole通过将视觉特征转换为通用潜在表示来实现此方法,但每个消息均以相同大小的稠密张量传输,无论内容如何。固定容量的稠密张量并不具备固定的有效信息密度:某些消息可能仅使用可用表征自由度的一小部分。这一观察表明通信通道具有显著可压缩性。我们通过对冻结的Vision Wormhole激活值拟合后置稀疏自编码器,评估了重建质量、下游任务性能、特征复用及标记级干预在九个推理基准上的表现。相较于原始float32传输,采用uint16索引/float16值的稀疏负载(每标记k=4个活跃系数)将传输字节数减少128倍。单次评估中,七项任务非AIME平均准确率从49.85%降至49.77%。拟合的4096维词典仅使用50个特征,任务级活跃集合的平均成对Jaccard相似度达0.906。这些测量结果证明了相对于原始传输方式的强大后置可压缩性,但尚未区分稀疏编码与位置选择、精度降低、低秩结构或SAE优化效果的增量贡献。结果推动开展匹配负载对比与动态适应信息使用的通信机制研究。

原文摘要 · Abstract (English)

Latent-space communication allows heterogeneous vision-language model agents to exchange continuous representations without serializing visual and reasoning states into text. Vision Wormhole realizes this approach by translating visual features into a universal latent representation that can be consumed by another model, but every message is transported as a dense tensor of the same size regardless of its content. A fixed-capacity dense tensor therefore need not have a fixed effective information density: some messages may use only a small fraction of the available representational degrees of freedom. This observation suggests that the communication channel may be substantially compressible. We study its redundancy by fitting a post-hoc sparse autoencoder to frozen Vision Wormhole activations and measuring reconstruction, downstream utility, feature reuse, and token-level interventions across nine reasoning benchmarks. Relative to the original float32 transport, a uint16-index/float16-value sparse payload with k=4 active coefficients per token reduces the transmitted bytes by 128x. In a single-run evaluation, the seven-task non-AIME mean accuracy changes from 49.85% to 49.77%. The fitted 4096-element dictionary uses only 50 features, and task-level active sets have a mean pairwise Jaccard similarity of 0.906. These measurements establish strong post-hoc compressibility relative to the original transport, but do not yet isolate the incremental contribution of sparse coding from position selection, reduced precision, low-rank structure, or SAE optimization effects. The results motivate matched-payload comparisons and communication mechanisms whose payload adapts to the information used by each message.

稀疏编码模型通信高效传输视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。