通过复用历史输出加速流式翻译重生成,显著降低延迟。
Self-Speculative Biased Decoding for Faster Re-Translation
- 利用前次输出作为推测草稿,减少重复计算。
- 单次前向传播验证草稿,仅在首次差异处恢复自回归解码。
- 适合对实时性要求高的机器翻译场景,无需额外训练。
大型语言模型虽能实现高质量机器翻译,但推理成本高、延迟大,难以支持实时翻译。重生成方法通过不断重译目标输出来适应输入增长,但存在大量冗余计算。本文提出自推测偏差解码(SSBD),一种无需调参的推理方法,利用流式翻译中的时间一致性,将模型先前输出作为推测草稿,通过轻量级偏置在单次前向传播中高效验证,并仅在首次分歧处恢复自回归解码。此外,引入仅显示遮蔽策略,隐藏不稳定的后缀以提升界面体验,同时保留其用于验证和可能采纳。实验表明,SSBD在保持相近翻译质量的前提下,相比标准重生成实现了显著加速,且无需架构改动、辅助模型或额外微调。
原文摘要 · Abstract (English)
Large language models achieve strong machine translation quality but incur high inference cost and latency, posing challenges for simultaneous translation. Re-translation provides a practical solution for off-the-shelf LLMs by repeatedly regenerating the target output as the source input grows, but it suffers from substantial redundant computation. We propose Self-Speculative Biased Decoding (SSBD), a simple and tuning-free inference method that accelerates re-translation by exploiting temporal coherence in streaming translation. SSBD reuses the model's previous output as a speculative draft for the updated input, verifies the draft efficiently in a single forward pass with a lightweight bias, and resumes autoregressive decoding only from the first divergence. We further introduce a display-only masking strategy that hides unstable suffixes from the user interface while retaining them in the draft for verification and potential acceptance. Experiments show that SSBD achieves substantial speedup over standard re-translation while maintaining comparable translation quality, without architectural changes, auxiliary models, or extra fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。