用智能调度让自部署大模型更省心省钱。
Efficient Multi-Model Orchestration for Self-Hosted Large Language Models
- 基于Kubernetes的动态调度框架，自动伸缩并智能路由请求。
- 实测比静态部署降低33%显卡成本，延迟降30%，成功率升21.6%。
- 适合需要私有化部署且追求性价比的团队或企业使用。
自部署大型语言模型正受到越来越多组织青睐，以保障隐私、控制成本并实现定制化。然而，本地部署面临显卡利用率低、任务路由难和系统可靠性差等挑战。我们提出Pick and Spin框架，使自托管LLM的编排具备可扩展性和经济性。该框架基于Kubernetes构建，集成统一的Helm部署系统、自适应的零缩放自动化机制，以及混合式路由模块——结合关键词启发式规则与轻量级DistilBERT分类器，平衡成本、延迟与准确率。我们在八个公开基准数据集上对Llama-3（90B）、Gemma-3（27B）、Qwen-3（235B）和DeepSeek-R1（685B）四款模型进行了评估，涵盖五种推理策略与两种路由变体，共计31,019个提示和163,720次推理。结果显示，与相同模型的静态部署相比，Pick and Spin实现了最高21.6%的成功率提升，延迟降低30%，每查询显卡成本下降33%。
原文摘要 · Abstract (English)
Self-hosting large language models (LLMs) is increasingly appealing for organizations seeking privacy, cost control, and customization. Yet deploying and maintaining in-house models poses challenges in GPU utilization, workload routing, and reliability. We introduce Pick and Spin, a practical framework that makes self-hosted LLM orchestration scalable and economical. Built on Kubernetes, it integrates a unified Helm-based deployment system, adaptive scale-to-zero automation, and a hybrid routing module that balances cost, latency, and accuracy using both keyword heuristics and a lightweight DistilBERT classifier. We evaluate four models, Llama-3 (90B), Gemma-3 (27B), Qwen-3 (235B), and DeepSeek-R1 (685B) across eight public benchmark datasets, with five inference strategies, and two routing variants encompassing 31,019 prompts and 163,720 inference runs. Pick and Spin achieves up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments of the same models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。