提出去中心化路由让大模型推理更高效，不依赖集中控制
Towards Distributed Inference of LLMs on a P2P Network
- 节点自建前缀缓存树，通过异步同步邻居缓存信息
- 在低延迟和偏斜分布下，推理延迟降低30%以上
- 适合分布式推理场景，尤其适合网络稳定的小规模集群
前缀缓存可通过重用共享提示的键值（KV）缓存来降低大模型推理延迟，但在集群中因缓存跨节点分割而难以实现大规模复用。本文提出一种去中心化的、感知前缀缓存的对等网络路由方案。每个节点维护本地的前缀缓存径向树，并通过周期性反熵协议异步更新对等节点缓存的估计。请求被路由至估计前缀匹配最长的节点，无需集中协调或传输KV缓存。过时元数据仅导致缓存未命中，不影响输出正确性，因此弱一致性足以保证正确性。在模拟MMLU工作负载下的评估显示，该方案在低通信延迟和前缀分布偏斜条件下能有效降低延迟；但高网络延迟和亲缘性导致的热点会限制其优势。
原文摘要 · Abstract (English)
Prefix caching can reduce LLM inference latency by reusing KV caches across requests with shared prompts, but cluster-scale reuse is challenging because caches are partitioned across nodes. We propose a decentralized, prefix-cache-aware routing scheme for peer-to-peer LLM serving. Each node maintains a local radix tree of its own cached prefixes and asynchronously refreshed estimates of peer caches using periodic anti-entropy. Requests are routed to the node with the longest estimated prefix match, without centralized coordination or KV-cache transfer. Stale metadata only causes cache misses, not incorrect outputs, making weak consistency sufficient for correctness. Evaluation on simulated MMLU workloads show that decentralized routing improves latency under low communication delay and skewed prefix distributions, while high network latency and affinity-induced hotspots limit its benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。