让大模型服务跑程序而非提示词，提升效率与灵活性。
Serve Programs, Not Prompts
- 用可运行的程序（LIPs）替代固定提示，支持动态调整生成逻辑。
- 系统通过虚拟缓存和两级调度，实现高GPU利用率与低延迟。
- 适合需要自定义推理流程或工具调用的复杂应用开发者。
当前的大语言模型（LLM）服务系统主要针对文本补全设计，面对日益复杂的应用时，其僵化的架构导致效率低下且难以适配。本文提出一种新架构，通过服务程序而非提示来解决该问题。这些程序称为大模型推理程序（LLM Inference Programs, LIPs），允许用户在运行时自定义令牌预测与键值缓存管理，并将工具执行等应用逻辑卸载至服务器。我们以名为Symphony的系统为例，它作为LIPs的操作系统，通过系统调用暴露模型计算能力，使用专用文件系统虚拟化KV缓存，并采用两级进程调度机制保障GPU效率。该系统有望开启更高效、可扩展的LLM应用生态。
原文摘要 · Abstract (English)
Current large language model (LLM) serving systems, primarily designed for text completion, are neither efficient nor adaptable for increasingly complex LLM applications due to their inflexible design. We propose a new LLM serving system architecture that serves programs instead of prompts to address this problem. These programs, called LLM Inference Programs (LIPs), allow users to customize token prediction and KV cache management at runtime and to offload parts of their application logic, such as tool execution, to the server. We describe an example of this architecture through a system named Symphony, which functions as an operating system for LIPs. Symphony exposes LLM model computations via system calls and virtualizes KV cache with a dedicated file system, while ensuring GPU efficiency with a two-level process scheduling scheme. Symphony has the potential to open the door to a more efficient and extensible ecosystem for LLM applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。