用字节级分解取代传统词嵌入，大幅减少参数量且保持性能。
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
- 通过字节与位置的克罗内克积分解实现固定编码器+可学习投影
- 在124M模型上验证损失低2.5%，训练步数减少43%且更抗拼写错误
- 适合追求轻量化、高效推理的部署场景，尤其对资源受限环境友好
大型语言模型需通过 |V|×d_model 的可训练嵌入表处理输入，消耗数亿至数十亿参数。本文提出克罗内克嵌入，一种确定性的字节级字符-位置因子分解方法，以固定编码器和单个可学习投影替代该嵌入表，兼容标准BPE分词器，在前沿规模下可消除91%–94%的输入侧可训练参数。实验显示：跨六种模型（135M–671B参数）的探测表明，传统嵌入将拼写变体聚类得远比形态相关词紧密；而克罗内克嵌入在嵌入层避免了此类聚类。在nanoGPT GPT-2 124M模型上，使用25亿个FineWeb-Edu令牌进行三组种子对比，克罗内克嵌入达到2.5±0.2%更低的验证损失（差距0.083±0.007 nats，约9%更低困惑度），所需训练步数仅为基线的1.43倍即达收敛。拼写鲁棒性测试中，克罗内克在110组干净/拼写错误配对中保持最高预测正确率55.5%（基线47.3%），提升8.2个百分点，KL下降7.6%，11类中有10类胜出或持平；生成测试显示，克罗内克能保留并重现字节级新串和拼写错误，而基线会遗忘。此外，基线嵌入范数随训练漂移，而克罗内克投影范数稳定在1.0附近，符合稳定表示目标。进一步提出运行时实时重构版本，仅用4.5MB字节缓冲区替代2.15GB嵌入表（词汇量131,072），每步开销0.01%–0.24%。字节级局部性带来权衡：语义相异但字节相似的词对（如compute/commute, nation/notion）被聚在一起，使消歧任务转移至早期注意力层。
原文摘要 · Abstract (English)
Large language models route every input through a learned embedding table of shape |V| x d_model, consuming hundreds of millions to billions of trainable parameters at frontier scale. We introduce Kronecker Embeddings, a deterministic byte-level character-position factorization that replaces this table with a fixed encoder and a single learned projection, compatible with standard BPE tokenizers, eliminating 91--94% of input-side trainable parameters at frontier scale. We provide five contributions. First, a cross-model probe across six LMs (135M-671B parameters) shows trained input embeddings cluster typographic variants of the probe word far more than morphological relatives; Kronecker escapes this clustering at the embedding layer. Second, a controlled three-seed comparison on nanoGPT GPT-2 124M over 2.5B tokens of FineWeb-Edu shows Kronecker reaching 2.5 +- 0.2% lower validation loss than the BPE-tied baseline (gap 0.083 +- 0.007 nats, ~9% lower perplexity), needing ~1.43x fewer steps to reach BPE's converged loss. Third, a spelling-robustness probe over 110 clean/typo pairs shows Kronecker preserves the top-1 prediction on 55.5% of pairs vs. 47.3% for BPE (+8.2 pp) and lowers KL by 7.6%, winning or tying in 10 of 11 categories; a generation probe shows Kronecker echoes byte-novel strings and typos through generation where BPE forgets them. Fourth, BPE embedding norm drifts during training while Kronecker projection norm stays near 1.0, consistent with a stable representational target. Fifth, an on-the-fly runtime variant reconstructs embeddings from a 4.5 MB byte buffer rather than a 2.15 GB table at vocabulary 131,072, with 0.01--0.24% step-time overhead. Byte-level locality has a tradeoff: byte-similar but semantically distant pairs (compute/commute, nation/notion) cluster together, shifting disambiguation to early attention layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。