研究量化对翻译模型推理效率的影响,发现分块策略与量化方式协同优化可显著提升性能。
Studying quantization trade-offs for efficient inference deployment in machine translation
- 结合文档分块与W4A8/W8A8量化,提升延迟-吞吐权衡表现
- 标准句级评估无法预测量化对长文本翻译的影响,真实效果差30%以上
- Hy-MT2抗量化能力强,EuroLLM在量化下质量骤降
在真实服务器环境中部署大语言模型面临挑战,系统需在低延迟下提供高质量响应。量化是降低内存占用、提升推理效率的常用方法,但其对延迟和吞吐量的影响常缺乏在受控编排工作负载下的评估。本文研究了两个翻译模型族(EuroLLM 和 Hy-MT2)在五种规模从1.7B到22B的模型上,使用单张A100或H100 GPU进行高效部署时的量化权衡。结果表明,结合文档分块策略与W4A8或W8A8量化,可在多种工作负载下显著改善延迟-吞吐帕累托曲线。此外,由于标准机器翻译基准依赖孤立句子,无法捕捉长上下文动态,我们引入了来自WMT24++的文档级评估,以分析分块策略在量化下的翻译质量影响。结果显示,标准段落级评估无法准确预测量化与长文本翻译之间的交互作用。尽管Hy-MT2在量化下保持鲁棒性,EuroLLM表现出强烈敏感性,所有考虑的量化格式下翻译质量均迅速下降。总体而言,推理效率与翻译质量的权衡不仅取决于量化格式,还受文本分块策略选择的影响。
原文摘要 · Abstract (English)
Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common approach to reduce the memory footprint and improve inference efficiency, yet its impact on latency and throughput is rarely evaluated under controlled, orchestration-level workloads. In this work we study the quantization trade-offs of two translation model families, EuroLLM \citep{martins2025eurollm} and Hy-MT2 \citep{zheng2026hy} across five models ranging from 1.7B to 22B for efficient deployment on a single A100 or H100 GPU. We demonstrate that combining a document-chunking strategy with W4A8 or W8A8 quantization improves the latency-throughput Pareto-curve under a wide range of workloads. Furthermore, since standard machine translation (MT) benchmarks rely on isolated sentences and fail to capture long-context dynamics, we introduce a document-level evaluation from WMT24++ to assess how text chunking strategies affect translation quality under quantization. Our results reveal that standard segment-level evaluation can fail to predict the interaction between quantization and long-context document translation. While Hy-MT2 remains robust under quantization, EuroLLM shows strong sensitivity and translation quality collapses rapidly for all considered quantization formats. Overall, our experiments show that the trade-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。