提出字节级分词压缩方法，解决长尾字符序列过长问题
Bit-level BPE: Below the byte boundary
- 将字符按比特级重组，避免字节级分词导致的序列膨胀
- 在中文日文韩文等语言上实现序列长度降低30%以上
- 适合需要高效处理多语言和表情符号的模型部署场景
大型语言模型中常见的子词分词会采用字节级回退以防止未登录词（OOV）。然而，将字符拆分为单个字节会显著增加中文、日文、韩文（CJK）等语言以及表情符号等字符多样性高的语境下的序列长度，导致训练与推理时计算开销上升。本文提出一种简单的无损压缩技术，可在不损失信息的前提下有效减少序列长度。
原文摘要 · Abstract (English)
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。