通过限制缓存大小,解决实时对话模型服务中的突然崩溃问题。
Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving
- 为每个会话的缓存设置固定窗口,防止内存耗尽
- 缓存上限使延迟变为负载的单调信号,避免突然失效
- 适用于需要稳定实时响应的语音交互系统
实时交互模型(如 Moshi、MiniCPM-o、Qwen-Omni)在每帧周期内需处理流式音频并按时响应,其键值(KV)缓存随对话持续增长且长期驻留。此模式隐藏严重故障风险:在真实全双工环境下,持续负载不会渐进降级,而是突然崩溃——从毫秒级延迟跃升至引擎停滞,且仅在20次运行中有6次失败,具有随机性且无明显异常指标。本文提出在引擎内设置KV缓存窗口,可彻底消除崩溃(两批测试中0/20对比14/20),使每帧延迟成为稳定的负载指示器,并支持在线准入控制器识别可调度并发数。首个一阶模型预测崩溃时间误差小于百分之几;质量探测显示,该窗口本身不降低生成质量,仅保留少量关键注意力锚点即可维持自由生成健康。所有实验均基于真实音频,在单张GPU上覆盖四个交互模型完成端到端验证。
原文摘要 · Abstract (English)
Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。