对比不同GPU下开源大模型的推理性能，助你选对模型部署
Deploying Open-Source Large Language Models: A performance Analysis
- 用vLLM库测试Mistral和LLaMa系列模型在多种显卡上的表现
- 明确不同模型在特定硬件下的吞吐量与延迟数据
- 适合想私有部署大模型的技术团队参考
自2022年11月ChatGPT发布以来，大型语言模型（LLMs）在开源社区也取得显著进展，众多开源权重模型陆续问世。然而，部署此类服务所需的资源要求往往不明确且难以预先评估。为解决此问题，我们在波尔多大学的Inria中心进行了大量测试。本文比较了不同规模模型（主要为Mistral和LLaMa系列）在不同可用GPU下的性能表现，采用vLLM这一专为优化模型推理设计的Python库。研究结果为希望部署大模型的个人及公共机构提供了关键性能参考，使其可根据自身硬件条件合理选择模型，从而推动大模型在各类应用场景中的落地与使用。
原文摘要 · Abstract (English)
Since the release of ChatGPT in November 2022, large language models (LLMs) have seen considerable success, including in the open-source community, with many open-weight models available. However, the requirements to deploy such a service are often unknown and difficult to evaluate in advance. To facilitate this process, we conducted numerous tests at the Centre Inria de l'Université de Bordeaux. In this article, we propose a comparison of the performance of several models of different sizes (mainly Mistral and LLaMa) depending on the available GPUs, using vLLM, a Python library designed to optimize the inference of these models. Our results provide valuable information for private and public groups wishing to deploy LLMs, allowing them to evaluate the performance of different models based on their available hardware. This study thus contributes to facilitating the adoption and use of these large language models in various application domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。