用持续状态缓存让大模型推理速度提升5.9倍，且延迟不随上下文增长
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
- 引入持久化键值缓存，新数据到来时增量更新，避免每次查询重算
- 在流式市场数据上实现最高5.9倍加速，查询延迟恒定不变
- 适合需要实时响应的连续流式任务，如金融行情分析
传统Transformer推理引擎按请求驱动，每次查询都需付出O(n)预填充开销。在数据持续到达的流式场景中，这种开销难以承受。本文提出以数据驱动的计算模型，基于持续会话：一个随新数据增量更新的持久键值缓存，使预填充脱离关键路径，查询延迟降为O(|q|)，与累积上下文大小无关。在此基础上，Flash Queries利用数据间隙空闲的GPU周期，提前预评估注册问题并返回缓存结果，这一机制在无状态引擎中因中间状态丢失而无法实现。结合多租户连续批处理调度器、单元预算准入机制和前缀感知分组预填充，数十个状态会话可共存于单张GPU上，同时保持完整的二次自注意力能力。在流式市场数据基准测试中，参考实现相较vLLM、SGLang、TensorRT-LLM、llama.cpp等引擎最高提速5.9倍，且查询延迟不随上下文积累而增加。
原文摘要 · Abstract (English)
Conventional transformer inference engines are request-driven, paying an O(n) prefill cost on every query. In streaming workloads, where data arrives continuously and queries probe an ever-growing context, this cost is prohibitive. We introduce a data-driven computational model centred on stateful sessions: a persistent KV cache advanced incrementally as new data arrives, so prefill is moved off the critical path and query latency becomes O(|q|), independent of accumulated context size. Building on this, Flash Queries reclaim idle GPU cycles between data arrivals to pre-evaluate registered questions and return cached answers before the user asks, a pattern that is structurally impossible in stateless engines because they discard intermediate state between requests. A multi-tenant continuous-batching scheduler with cell-budget admission and prefix-aware grouped prefill lets dozens of stateful sessions coexist on a single GPU while preserving full quadratic self-attention. On streaming market-data benchmarks the reference implementation achieves up to 5.9x speedup over conventional inference engines (vLLM, SGLang, TensorRT-LLM, llama.cpp), holding query latency constant as accumulated context grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。