通过分阶段流式输出，显著降低多智能体推理的首字延迟。
Staircase Streaming for Low-Latency Multi-Agent Inference
- 在接收部分中间结果时即开始生成最终回答，减少等待时间。
- 实验显示首字延迟最高降低93%，同时保持响应质量。
- 适合对响应速度要求高的实时对话系统应用。
大型语言模型（LLMs）的进展为利用多个LLM的集体能力开辟了新方向。例如混合智能体方法通常需要额外的推理步骤生成中间输出，再据此生成最终响应。尽管多智能体推理能提升响应质量，但会显著增加首次词延迟（TTFT），影响低延迟应用场景和用户体验。为此，我们提出阶梯式流式推理（staircase streaming），无需等待前步完整中间输出，即可在接收到部分中间结果时立即开始生成最终响应。实验表明，该方法可将TTFT降低高达93%，同时保持响应质量。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) opened up new directions for leveraging the collective expertise of multiple LLMs. These methods, such as Mixture-of-Agents, typically employ additional inference steps to generate intermediate outputs, which are then used to produce the final response. While multi-agent inference can enhance response quality, it can significantly increase the time to first token (TTFT), posing a challenge for latency-sensitive applications and hurting user experience. To address this issue, we propose staircase streaming for low-latency multi-agent inference. Instead of waiting for the complete intermediate outputs from previous steps, we begin generating the final response as soon as we receive partial outputs from these steps. Experimental results demonstrate that staircase streaming reduces TTFT by up to 93% while maintaining response quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。