用推测解码加速支付场景大模型，零成本提升吞吐量30%以上。
Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models
- 采用推测解码技术，在不增加硬件下优化推理效率。
- gamma=3时吞吐提升22%-49%，延迟降低18%-33%。
- 输出质量不变，单卡性能媲美双卡NIM，节省50%算力成本。
我们评估了在支付业务大模型Commerce Agent中使用EAGLE3进行推测解码的推理优化效果，该模型基于微调后的llama3.1-nemotron-nano-8B-v1。在相同2xH100硬件环境下，对比vLLM与NVIDIA NIM，覆盖40种配置（推测令牌数gamma=3, gamma=5；并发度1-32；采样温度0, 0.5）。关键发现：(1) gamma=3实现22%-49%吞吐提升和18%-33%延迟降低，零额外硬件成本；(2) gamma=3下接受率稳定在约35.5%；(3) gamma=5收益递减，接受率约25%；(4) LLM-as-Judge评估显示输出质量完全保留；(5) 单H100上推测解码性能匹配或超越双H100 NIM，实现50% GPU成本削减。
原文摘要 · Abstract (English)
We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on prior work (NEMO-4-PAYPAL) that reduced latency and cost through domain-specific fine-tuning, we benchmark EAGLE3 via vLLM against NVIDIA NIM on identical 2xH100 hardware across 40 configurations spanning speculative token counts (gamma=3, gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5). Key findings: (1) gamma=3 achieves 22-49% throughput improvement and 18-33% latency reduction at zero additional hardware cost; (2) acceptance rates remain stable at approximately 35.5% for gamma=3 across all conditions; (3) gamma=5 yields diminishing returns (approximately 25% acceptance rate); (4) LLM-as-Judge evaluation confirms fully preserved output quality; and (5) speculative decoding on a single H100 matches or exceeds NIM on two H100s, enabling 50% GPU cost reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。