让不同模型通过视觉隐空间通信,提速且不丢信息。
The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
- 用视觉编码器做跨模型隐状态传递,无需成对翻译器。
- 在9个推理任务上平均准确率提升,端到端速度更快。
- 适合多模型协作场景,尤其异构视觉语言模型间通信。
基于大语言模型的多智能体系统虽能实现高级协同推理,但受限于离散文本通信带来的运行开销与信息量化损失。现有隐状态传输方法或假设发送方与接收方架构相同,或依赖成对训练的翻译器,难以扩展至不同模型族间的异构场景。本文将视觉语言模型(VLM)面向自然图像的视觉接口重新定义为连续通信通道,提出 extbf{Vision Wormhole}:一个通用视觉编解码器将推理轨迹映射至共享连续参考空间,并注入接收端视觉路径,实现跨架构隐状态传输,无需每对模型专属翻译器。该框架采用中心辐射拓扑,将对齐复杂度从$O(N^2)$降至$O(N)$,并通过无标签师生蒸馏训练,无需并行隐藏状态监督。在Qwen-VL、Gemma、SmolVLM2、LFM2.5-VL等异构VLM家族及九个推理基准上的实验表明,Vision Wormhole在多数设置下显著降低端到端时延,并带来正向宏观平均Δ-准确率提升。
原文摘要 · Abstract (English)
Multi-Agent Systems (MAS) powered by Large Language Models have unlocked advanced collaborative reasoning, yet they remain bottlenecked by discrete text communication, which imposes runtime overhead and information quantization loss. While latent state transfer offers an alternative, existing approaches either assume homogeneous sender--receiver architectures or rely on pair-specific learned translators, limiting scalability across diverse model families with disjoint manifolds. We reconceptualize the visual interface of Vision-Language Models (VLMs), trained for natural images, as a continuous communication channel between heterogeneous agents, and instantiate this idea as the \textbf{Vision Wormhole}: a Universal Visual Codec maps reasoning traces into a shared continuous reference space and injects them into the receiver's visual pathway, yielding cross-architecture latent state transfer without per-pair translators. The framework adopts a hub-and-spoke topology that reduces alignment complexity from $O(N^2)$ to $O(N)$, and is trained by label-free teacher--student distillation against the text channel, requiring no parallel hidden-state supervision. Extensive experiments across heterogeneous VLM families (Qwen-VL, Gemma, SmolVLM2, LFM2.5-VL) and nine reasoning benchmarks show that the Vision Wormhole reduces end-to-end wall-clock time across most evaluated settings and yields positive macro-average $Δ$-accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。