KevlarFlow让大模型服务在硬件故障时仍能快速恢复，提升稳定性。
Towards Resiliency in Large Language Model Serving with KevlarFlow
- 分步初始化模型并行，故障时无需全量重载
- 故障下平均延迟降低3.1倍，首令牌延迟提升378倍
- 适合高可用大模型部署，如AI云服务场景
大规模语言模型（LLM）服务系统仍存在根本性脆弱性，超大规模集群中的频繁硬件故障会引发软件层的严重服务中断。现有恢复机制极为缓慢，资源重新初始化和模型权重加载常需长达10分钟。本文提出KevlarFlow，一种容错服务架构，旨在弥合硬件不可靠性与服务可用性之间的差距。该架构采用：1）解耦的模型并行初始化、2）动态流量重路由、3）后台KV缓存复制，以在部分故障情况下维持高吞吐。评估显示，相比当前最先进的LLM服务系统，KevlarFlow将平均恢复时间（MTTR）降低20倍；在故障条件下，平均延迟改善3.1倍，p99延迟改善2.8倍，平均首令牌时间（TTFT）提升378.9倍，p99 TTFT提升574.6倍，且运行开销可忽略不计。
原文摘要 · Abstract (English)
Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow, a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。