根据用户认知负荷动态调节大模型输出速度，节省算力。
Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving
- 通过推断内容认知负荷，实时调整生成文本的流速。
- 实验显示算力消耗降低，且流速仍高于正常阅读速度。
- 适合高并发场景下优化云服务资源分配。
由大语言模型（LLMs）驱动的生成式对话界面通常以计算预算决定的速率逐词流式输出，忽视了真实的人类阅读速度及内容带来的认知负荷。这种不匹配常导致计算资源浪费。例如，在云端服务中，输出速度超过用户阅读能力时，会无谓消耗算力，影响其他用户的响应延迟，尤其在高峰期更为明显。为此，我们提出一种自适应流式方法，基于推断的认知负荷实时动态调整LLM输出节奏。该方法评估流式内容的认知负荷，在复杂或信息密集段落主动放慢输出，从而释放计算资源供其他用户使用。我们基于众包用户研究收集的数据，构建统计模型并进行分析与仿真。结果表明，该方法能有效降低计算消耗，同时保持流速显著高于用户正常阅读速度。
原文摘要 · Abstract (English)
Generative conversational interfaces powered by large language models (LLMs) typically stream output token-by-token at a rate determined by computational budget, often neglecting actual human reading speeds and the cognitive load associated with the content. This mismatch frequently leads to inefficient use of computational resources. For example, in cloud-based services, streaming content faster than users can read appears unnecessary, resulting in wasted computational resources and potential delays for other users, particularly during peak usage periods. To address this issue, we propose an adaptive streaming method that dynamically adjusts the pacing of LLM streaming output in real-time based on inferred cognitive load. Our approach estimates the cognitive load associated with streaming content and strategically slows down the stream during complex or information-rich segments, thereby freeing computational resources for other users. We conducted a statistical analysis and simulation based on a statistical model derived from data collected in a crowdsourced user study across various types of LLM-generated content. Our results show that this adaptive method can effectively reduce computational consumption while largely maintaining streaming speed above user's normal reading speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。