对比FastAPI与Triton在医疗AI推理中的性能，提出安全高效的混合部署方案。
Scalable and Secure AI Inference in Healthcare: A Comparative Benchmarking of FastAPI and Triton Inference Server on Kubernetes
- 用Kubernetes部署DistilBERT模型，对比FastAPI与Triton的推理表现。
- Triton在单张T4 GPU上实现780请求/秒吞吐量，是FastAPI的近两倍。
- 混合架构兼顾安全与性能，适合医疗等高合规场景的生产部署。
高效可扩展的机器学习模型部署是现代生产环境的必备条件，尤其在医疗、制药等受监管领域。系统需平衡实时临床决策支持的低延迟、医疗记录批量处理的高吞吐，以及严格遵守如HIPAA等数据隐私标准的要求。本文对两种主流部署方式进行了严谨基准测试：基于Python的轻量级REST服务FastAPI，以及专为高性能推理设计的NVIDIA Triton Inference Server。在医疗AI参考架构下，我们将DistilBERT情感分析模型部署于Kubernetes，控制实验条件下测量中位数（p50）和尾部延迟（p95），以及吞吐量。结果表明存在明显权衡：FastAPI在单请求负载下开销更低，p50延迟为22毫秒；而Triton通过动态批处理实现更优扩展性，在单张NVIDIA T4 GPU上达到780请求/秒吞吐量，接近基线的两倍。此外，我们评估了混合架构——使用FastAPI作为敏感健康信息脱敏的安全网关，Triton负责后端推理。该研究验证了混合模式是企业级临床AI的最佳实践，并提供了一个安全、高可用部署的蓝图。
原文摘要 · Abstract (English)
Efficient and scalable deployment of machine learning (ML) models is a prerequisite for modern production environments, particularly within regulated domains such as healthcare and pharmaceuticals. In these settings, systems must balance competing requirements, including minimizing inference latency for real-time clinical decision support, maximizing throughput for batch processing of medical records, and ensuring strict adherence to data privacy standards such as HIPAA. This paper presents a rigorous benchmarking analysis comparing two prominent deployment paradigms: a lightweight, Python-based REST service using FastAPI, and a specialized, high-performance serving engine, NVIDIA Triton Inference Server. Leveraging a reference architecture for healthcare AI, we deployed a DistilBERT sentiment analysis model on Kubernetes to measure median (p50) and tail (p95) latency, as well as throughput, under controlled experimental conditions. Our results indicate a distinct trade-off. While FastAPI provides lower overhead for single-request workloads with a p50 latency of 22 ms, Triton achieves superior scalability through dynamic batching, delivering a throughput of 780 requests per second on a single NVIDIA T4 GPU, nearly double that of the baseline. Furthermore, we evaluate a hybrid architectural approach that utilizes FastAPI as a secure gateway for protected health information de-identification and Triton for backend inference. This study validates the hybrid model as a best practice for enterprise clinical AI and offers a blueprint for secure, high-availability deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。