提出LLM推理优化的三维度架构,系统梳理路由与资源管理的协同设计。
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
- 构建工作负载-路由器-资源池三维框架,统一调度、算力与安全策略
- 覆盖21个研究方向,基于实测数据明确已解决与待突破的交叉点
- 适合大规模部署者、系统优化工程师及多模态代理研发人员参考
vLLM语义路由项目在过去一年中推出系列成果:(1)核心路由机制——信号驱动路由、上下文长度池路由、路由器性能工程、策略冲突检测、低延迟嵌入模型、类别感知语义缓存、用户反馈驱动的路由自适应、幻觉检测,以及用于隐私与越狱防护的分层内容安全分类;(2)集群优化——集群配置与能效分析;(3)智能体与多模态路由——多模态智能体路由、工具选择、CUA安全、多轮上下文记忆与安全;(4)治理与标准——推理路由协议与多提供方API扩展。这些工作分别解决特定问题,但彼此依赖:集群配置依赖路由策略,路由策略又受工作负载组合影响,而组织正逐步采用智能体与多模态应用。本文提炼出工作负载-路由器-资源池(WRP)架构,一个面向大模型推理优化的三维框架。工作负载定义服务类型(聊天/智能体、单轮/多轮、热/冷、预填充密集/解码密集);路由器决定请求分发方式(静态语义规则、在线强化学习自适应、基于强化学习的模型选择、质量感知级联);资源池定义推理运行位置(同构/异构GPU、预填充/解码分离、KV缓存拓扑)。将过往成果映射至3×3的WRP交互矩阵,识别已覆盖与未覆盖的交叉领域,并提出21项具体研究方向,均基于前期测量结果,按成熟度分级,从工程可落地到开放研究。
原文摘要 · Abstract (English)
Over the past year, the vLLM Semantic Router project has released a series of work spanning: (1) core routing mechanisms -- signal-driven routing, context-length pool routing, router performance engineering, policy conflict detection, low-latency embedding models, category-aware semantic caching, user-feedback-driven routing adaptation, hallucination detection, and hierarchical content-safety classification for privacy and jailbreak protection; (2) fleet optimization -- fleet provisioning and energy-efficiency analysis; (3) agentic and multimodal routing -- multimodal agent routing, tool selection, CUA security, and multi-turn context memory and safety; (4) governance and standards -- inference routing protocols and multi-provider API extensions. Each paper tackled a specific problem in LLM inference, but the problems are not independent; for example, fleet provisioning depends on the routing policy, which depends on the workload mix, shifting as organizations adopt agentic and multimodal workloads. This paper distills those results into the Workload-Router-Pool (WRP) architecture, a three-dimensional framework for LLM inference optimization. Workload characterizes what the fleet serves (chat vs. agent, single-turn vs. multi-turn, warm vs. cold, prefill-heavy vs. decode-heavy). Router determines how each request is dispatched (static semantic rules, online bandit adaptation, RL-based model selection, quality-aware cascading). Pool defines where inference runs (homogeneous vs. heterogeneous GPU, disaggregated prefill/decode, KV-cache topology). We map our prior work onto a 3x3 WRP interaction matrix, identify which cells we have covered and which remain open, and propose twenty-one concrete research directions at the intersections, each grounded in our prior measurements, tiered by maturity from engineering-ready to open research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。