通过字符级多头输出，让大词汇量模型更快更省资源。
SpeLLM: Character-Level Multi-Head Decoding
- 用多个独立线性头并行预测字符，解耦输入输出词表
- 四款模型平均提速5.1%，下游任务表现相当
- 适合需要支持小语种或长尾领域的场景
扩大语言模型词表常用于减少输入序列长度，缓解注意力机制的二次方开销。然而现有架构存在关键瓶颈：输出投影层随词表规模线性增长，导致大规模扩展不可行。本文提出SpeLLM，通过将输入与输出词表解耦，采用k个输出头并行预测字符级字符串。每个线性头独立预测一个字符，从而以更小的头实现更大的输出空间。我们提出一种自蒸馏方法，将标准LLM转换为SpeLLM。在四个预训练模型上的实验表明，其SpeLLM变体在下游任务上保持竞争力，且各模型平均运行时间降低5.1%。该方法为降低大模型成本、增强对低资源语言和领域支持提供了可行路径。
原文摘要 · Abstract (English)
Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this procedure: the output projection layer scales linearly with vocabulary size, rendering substantial expansion impractical. We propose SpeLLM, a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads. In SpeLLM, each of the $k$ linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space using smaller, independent linear heads. We present a self-distillation approach for converting a standard LLM to a SpeLLM. Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1% on average across models. Our approach provides a potential avenue for reducing LLM costs, while increasing support for underrepresented languages and domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。