提出查询迁移新范式,显著降低跨GPU注意力通信开销。
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
- 将查询而非缓存迁移,利用压缩降低传输量至约1KB
- 实测在多节点H100集群上,延迟降低至数十微秒级
- 适用于多种大模型架构,只需测量两个关键参数即可适配
前沿大模型使用稀疏注意力索引器,按查询选择少量可重用的键值缓存块。当代码库规模超过单GPU容量时,缓存需跨实例分片,导致查询与所选块位于不同GPU,引发跨实例注意力计算。传统方案是移动缓存块,而本文提出反向思路:将每个令牌的键值压缩为窄向量,使查询行仅约1KB,小于其访问的块,因此路由查询常比移动缓存更高效。我们在真实多节点H100集群上评估跨实例的多头潜在注意力(MLA),构建了拓扑感知成本模型(包含探测/传输/计算/返回/合并五阶段)和闭式路由/获取/本地判断条件,系数通过实际IBGDA测量,在批量往返中预测精度达~7%。推理时,通过探测延迟而非峰值带宽选择传输路径,将缓存移动开销(连续块约3毫秒重适配或分散收集)替换为数十微秒级往返,适用于当前DeepSeek-V3.2、V4及GLM-5.1等支持压缩或稀疏选择的架构。扩展至新架构仅需测量两系数:路由负载与移动缓存成本。
原文摘要 · Abstract (English)
Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk. Agentic workloads hammer it: many sub-agents query one large codebase, reusing the same blocks. When that corpus outgrows one GPU it is partitioned across instances, so a query and the blocks it selects often sit on different GPUs: answering it means attention across instances. The reflex of prior cross-instance KV systems is to move the cache: pull the selected blocks to the requester. Multi-head Latent Attention inverts the arithmetic, compressing each token's key and value into one narrow vector, so a routed query row is only ~1 KB, smaller than the chunk it attends; routing the query is then often cheaper than moving the cache. Which primitive wins, over which fabric and request shape, is uncharted, least of all on device-initiated RDMA that makes per-request cross-node transfers cheap. We characterize cross-instance MLA attention on a real multi-node H100 cluster, distilling two reusable artifacts: a topology-aware cost model (probe / transfer / compute / return / merge) and a closed-form route/fetch/local predicate, whose constants we measure on real IBGDA, where the model tracks batched round-trips to within ~7%. At decode it routes the query, trading the cost of moving the cache (a ~3 ms re-adaptation splice for a contiguous chunk, or a scattered gather under selection) for a tens-of-microsecond round trip, and picks the fabric by probe latency, not peak bandwidth. We instantiate the cost model and predicate for MLA, but neither is MLA-specific: they apply wherever compression or sparse selection shrinks attention to small chunks (DeepSeek-V3.2, V4, and GLM-5.1 today). Extending them to a new architecture requires measuring just two coefficients: the routed payload and fetch's move-the-cache cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。