让语言模型一次生成多个字节，提速且不降效果。
Dynamic Multi-Byte Prediction With Hierarchical Language Models
- 用可变长度窗口匹配层级模型的隐含分段，实现并行生成。
- 设计新注意力掩码，保证并行生成时因果关系正确。
- 在多种任务上兼顾速度与效果，适合追求推理效率的场景。
字节级层级语言模型近年来成为子词分词模型的有力替代方案。然而，逐字节生成仍是推理速度的瓶颈。为此，我们提出多字节预测（MBP），通过并行生成多个字节显著提升推理速度，性能损失极小且无需额外参数。MBP基于多标记预测范式，引入两项关键创新：一是采用与层级语言模型隐含分段对齐的可变长度预测窗口；二是设计新型注意力掩码机制，实现并行字节生成的同时保持因果性。实验表明，多字节预测在生成任务、指令遵循、问答、摘要和机器翻译中均达到性能与推理吞吐量之间的帕累托最优平衡。
原文摘要 · Abstract (English)
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。