用混合精度优化GPU加速大模型推理，实现64倍提速且不丢精度。
GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference
- 采用混合精度策略，关键运算保浮点32位，线性层用半精度
- 单样本延迟低于10毫秒，内存减少63%，速度比CPU快64.4倍
- 适合需要低延迟部署的AI系统开发者参考
本文提出基于NVIDIA TensorRT的GPU加速推理管线，对BERT-base（110M参数）和GPT-2（124M参数）进行混合精度优化。在批处理大小1至32、序列长度32至512的范围内评估。系统相较CPU基线实现最高64.4倍加速，单样本推理延迟低于10毫秒，内存占用降低63%。引入混合精度策略：对softmax与层归一化等数值敏感操作保留FP32，线性层采用FP16。该方法保持高数值保真度（余弦相似度≥0.9998），避免NaN不稳定。系统为模块化容器化设计，支持超过360种配置的可复现基准测试。在NVIDIA A100上跨GPU验证显示，FP16加速比稳定在1.84x至2.00x之间，数值行为一致。下游SST-2任务测试表明精度无下降。WikiText-2验证显示，全FP16随机输入会低估NaN风险达6倍，而混合策略实现0% NaN发生率，余弦相似度≥0.9998。结果详尽刻画了不同GPU架构下的性能与精度权衡，为低延迟环境部署提供实用指导。
原文摘要 · Abstract (English)
This paper presents the design and evaluation of a GPU-accelerated inference pipeline for transformer models using NVIDIA TensorRT with mixed-precision optimization. We evaluate BERT-base (110M parameters) and GPT-2 (124M parameters) across batch sizes from 1 to 32 and sequence lengths from 32 to 512. The system achieves up to 64.4x speedup over CPU baselines, sub-10 ms latency for single-sample inference, and a 63 percent reduction in memory usage. We introduce a hybrid precision strategy that preserves FP32 for numerically sensitive operations such as softmax and layer normalization, while applying FP16 to linear layers. This approach maintains high numerical fidelity (cosine similarity >= 0.9998 relative to baseline outputs) and eliminates NaN instability. The pipeline is implemented as a modular, containerized system that enables reproducible benchmarking across more than 360 configurations. Cross-GPU validation on an NVIDIA A100 shows consistent FP16 speedup ratios between 1.84x and 2.00x, along with stable numerical behavior. Downstream evaluation on SST-2 demonstrates no accuracy degradation under hybrid precision. Validation on WikiText-2 shows that random inputs underestimate NaN instability by up to 6x for full FP16, while confirming the robustness of the hybrid approach (0.0 percent NaN, cosine similarity >= 0.9998). These results provide a detailed characterization of performance and accuracy trade-offs across GPU architectures and offer practical guidance for deploying transformer models in latency-critical environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。