为低延迟设备端AI服务设计可完整恢复的执行状态胶囊机制。
Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving
- 用图边界封装完整执行状态，支持快照、恢复、分支和回滚。
- 在RTX 5090上恢复延迟低于1毫秒，16k token时TTFT提升27倍。
- 适合交互式AI代理、语音系统等需频繁重置的实时场景。
主流大模型服务系统通过分页或基数键值（KV）缓存复用前缀计算，适用于高吞吐、高并发场景，但仅管理单一位置片段——即KV缓存。本文研究相反场景：低延迟、小批量、设备端物理AI服务，此类场景中交互式大模型代理、语音系统与机器人策略需在严格响应预算内反复分支、重置、中断与重新进入。为此，提出执行状态胶囊机制，可在已提交边界处对完整可恢复状态进行图绑定的快照与恢复。FlashRT是白盒后端运行时内核，其经评估的NVIDIA CUDA后端在连续静态缓冲区上运行捕获的图计划，无块表间接寻址。由于活跃状态为一组命名缓冲区，胶囊可对整个执行边界（包括KV、循环状态、卷积状态、MTP状态及元数据）进行快照、恢复、分叉或回滚。这将复用从令牌地址级的KV片段提升至图边界级的执行状态边界。在RTX 5090上，胶囊恢复在存储状态层面字节精确，在贪婪解码下令牌完全一致；仅保留KV缓存的消融实验出现偏差，表明循环状态具有关键作用。GPU驻留的快照与恢复耗时亚毫秒，与冷预填充相比，TTFT加速从2000令牌时的3.9倍提升至16000令牌时的27倍。在Jetson AGX Thor和DGX Spark上同样保持正确性与结构特性。胶囊并非替代高吞吐KV缓存服务，而是为显式执行状态复用定义了一种以低延迟优先的互补服务范式。
原文摘要 · Abstract (English)
Mainstream LLM serving systems reuse prefix work mainly through paged or radix key-value (KV) caches. This is highly effective for high-throughput, high-concurrency serving, but it manages only one positional fragment of execution state: the KV cache. We study the opposite regime: low-latency, small-batch, on-device physical-AI serving, where interactive LLM agents, speech systems, and robot policies repeatedly branch, reset, interrupt, and re-enter under tight responsiveness budgets. We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state at a committed boundary. FlashRT is a white-box, backend-facing kernel runtime whose evaluated NVIDIA CUDA backend runs captured graph plans over contiguous static buffers with no block-table indirection. Because the live state is a closed set of named buffers, a capsule can snapshot, restore, fork, or roll back the whole execution boundary, including KV, recurrent state, convolution state, MTP state, and metadata. This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries. On an RTX 5090, capsule restore is byte-exact at the stored-state level and token-identical under greedy decode. A KV-only ablation diverges, showing that recurrent state is load-bearing. GPU-resident snapshot and restore are sub-millisecond, and TTFT speedup over cold prefill grows from 3.9x at 2k tokens to 27x at 16k tokens. On Jetson AGX Thor and DGX Spark, the same correctness and structural properties hold. Capsules are not a replacement for high-throughput KV-cache serving; they define a complementary latency-first serving point for explicit execution-state reuse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。