用选择性共享键值对实现高效大模型通信
KVComm: Enabling Efficient LLM Communication through Selective KV Sharing
- 通过注意力重要性筛选关键键值对，避免信息冗余
- 仅传输30%层的键值对，性能接近直接合并输入的上限
- 适合需要低延迟多模型协作的系统设计
大型语言模型在多智能体系统中的部署日益增多，有效的模型间通信至关重要。现有协议或依赖自然语言（成本高、易丢失信息），或依赖隐藏状态（存在信息集中偏差、效率低）。为此，我们提出KVComm，一种通过选择性共享键值对实现高效通信的新框架。该方法利用键值对中丰富的信息，规避隐藏状态的缺陷。我们引入基于注意力重要性得分与高斯先验的层级选择策略，识别最具信息量的键值对用于通信。在多种任务和模型组合上的实验表明，KVComm在仅传输30%层的键值对情况下，性能接近将输入直接合并至单一模型的上界方法。本研究揭示了键值对作为模型间通信有效媒介的潜力，为可扩展、高效的多智能体系统铺平道路。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in multi-agent systems, where effective inter-model communication is crucial. Existing communication protocols either rely on natural language, incurring high inference costs and information loss, or on hidden states, which suffer from information concentration bias and inefficiency. To address these limitations, we propose KVComm, a novel communication framework that enables efficient communication between LLMs through selective sharing of KV pairs. KVComm leverages the rich information encoded in the KV pairs while avoiding the pitfalls of hidden states. We introduce a KV layer-wise selection strategy based on attention importance scores with a Gaussian prior to identify the most informative KV pairs for communication. Extensive experiments across diverse tasks and model pairs demonstrate that KVComm achieves comparable performance to the upper-bound method, which directly merges inputs to one model without any communication, while transmitting as few as 30\% of layers' KV pairs. Our study highlights the potential of KV pairs as an effective medium for inter-LLM communication, paving the way for scalable and efficient multi-agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。