用2字节统一编码提升多语言语音识别的效率
BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition
- 基于UTF-16设计新分词器，统一使用2字节表示多数文字
- 中文等语言token数减少最多10.4%，解码迭代降10.3%
- 适合需要高效推理和低内存占用的多语言语音识别场景
多语言自动语音识别（ASR）需要能覆盖多种文字系统的分词方法。现有基于UTF-8的字节级BPE（BBPE）虽具备语言无关性且支持完整Unicode，但其变长编码导致中文、日文、韩文（CJK）等非拉丁文字序列过长，增加计算负载与内存开销。本文提出基于UTF-16的BBPE16分词器，将多数现代文字统一表示为2字节代码单元。该方法在保持语言无关特性的基础上，显著提升跨语言分词共享能力。在单语、双语、三语ASR及多语言持续学习设置中，BBPE16实现相当或更优的准确率；对中文，最大可减少10.4%的token数量，解码迭代降低最多10.3%。该改进加速了微调与推理过程，降低内存占用，是多语言ASR中实用的分词选择。
原文摘要 · Abstract (English)
Multilingual automatic speech recognition (ASR) requires tokenization that efficiently covers many writing systems. Byte-level BPE (BBPE) using UTF-8 is widely adopted for its language-agnostic design and full Unicode coverage, but its variable-length encoding inflates token sequences for non-Latin scripts, such as Chinese, Japanese, and Korean (CJK). Longer sequences increase computational load and memory use. We propose BBPE16, a UTF-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit. BBPE16 preserves BBPE's language-agnostic properties while substantially improving cross-lingual token sharing. Across monolingual, bilingual, and trilingual ASR, and in a multilingual continual-learning setup, BBPE16 attains comparable or better accuracy; for Chinese, it reduces token counts by up to 10.4% and lowers decoding iterations by up to 10.3%. These reductions speed up fine-tuning and inference and decrease memory usage, making BBPE16 a practical tokenization choice for multilingual ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。