为机器人实时任务优化的LLM服务系统，提升响应效率。
TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications
- 分段生成与调度机制，利用机器人规划与执行的冗余时间
- 时间效用最高提升1.97倍，等待时间减少84%
- 适合对响应速度敏感的无人机、机器人控制场景
大型语言模型（如GPT-4和Llama3）已能理解复杂指令并处理多样化任务，推动其在无人机和机器人控制中的应用。然而，现有LLM服务系统普遍采用先到先服务（FCFS）批处理机制，难以满足机器人应用的时间敏感需求。为此，本文提出TimelyLLM系统，支持多个机器人代理的时序敏感请求。该系统引入分段生成与调度新机制，充分利用机器人任务规划与执行阶段之间的冗余时间。我们在一个广泛使用的LLM服务框架上实现了TimelyLLM，并在多种机器人应用场景中进行评估。结果表明，该系统可将时间效用最高提升1.97倍，总体等待时间减少84%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) such as GPT-4 and Llama3 can already comprehend complex commands and process diverse tasks. This advancement facilitates their application in controlling drones and robots for various tasks. However, existing LLM serving systems typically employ a first-come, first-served (FCFS) batching mechanism, which fails to address the time-sensitive requirements of robotic applications. To address it, this paper proposes a new system named TimelyLLM serving multiple robotic agents with time-sensitive requests. TimelyLLM introduces novel mechanisms of segmented generation and scheduling that optimally leverage redundancy between robot plan generation and execution phases. We report an implementation of TimelyLLM on a widely-used LLM serving framework and evaluate it on a range of robotic applications. Our evaluation shows that TimelyLLM improves the time utility up to 1.97x, and reduces the overall waiting time by 84%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。