让大模型在用户说话时提前生成回复，大幅降低语音交互延迟。
PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction
- 用户说话时预生成回复候选，实现输入端推测解码
- 实测可将响应延迟降低约2倍，且额外计算开销极小
- 适合资源有限的设备上实时语音助手场景
大型语言模型（LLMs）广泛应用于实时语音聊天场景，通常与文本转语音（TTS）系统结合生成音频回应。然而其庞大的规模常导致用户输入结束到音频输出开始之间存在明显延迟，影响体验，尤其在算力有限的消费级硬件上更为突出。我们发现该延迟主要由LLM生成首句所需时间主导，而TTS系统需以句子为单位逐句合成音频。为此，我们提出预测生成（PredGen）框架，通过输入端推测解码缓解甚至消除这一延迟。PredGen在用户说话时即生成候选回复，使TTS可几乎无延迟启动。在Lmsys和MT-Bench数据集上的模拟实验表明，该方法在多种使用场景下可有效将延迟降低约2倍，且仅引入微小额外计算开销——这部分计算本就可能闲置。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely used in real-time voice chat applications, typically in combination with text-to-speech (TTS) systems to generate audio responses. However, their large size often leads to noticeable latency between the end of user input and the start of audio output, resulting in suboptimal user experiences. This latency is particularly evident when LLMs are deployed as single-user voice assistants on consumer-grade hardware with limited computing capacity. We discovered that this latency is primarily dominated by the time it takes for the LLMs to generate the first sentence, which is required as input by the TTS systems that synthesize audio responses on a sentence-by-sentence basis. To address this bottleneck, we propose Predictive Generation (PredGen), a novel framework that mitigates-or even eliminates-this delay through speculative decoding at input time. PredGen generates candidate responses while the user is still speaking, enabling the system to begin TTS processing with minimal delay. Simulated experiments on the Lmsys and MT-Bench datasets show that the proposed method can effectively reduce the latency by around 2x across a wide range of use cases, while incurring only minimal additional computation cost at input time-computation that would otherwise go unused.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。