arXiv:2510.15517cs.CL2025-10EMNLP

动态分组字符,让大模型更高效地处理罕见词。

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE

  • 用已有BPE结构动态分组字符,无需额外模型。
  • 在多个语言上表现优于或持平现有方法,词汇量小。
  • 适合追求高效、通用的自然语言处理研究者。

子词分词方法如字节对编码(BPE)因兼顾词汇紧凑性与表征能力,被广泛用于大型语言模型。然而,它们在表示稀有词时效率低下,且需要庞大的嵌入矩阵。字符级模型虽可缓解这些问题,但在基于Transformer的架构中引入性能瓶颈。近期的层次化模型尝试融合两种范式,通过将字符分组为块来实现,但现有分块策略或受限于空格,仅适用于特定语言,或需依赖额外模型带来新依赖。本文提出一种动态字符分组方法,利用现有BPE分词结构,无需额外模型。通过在BPE token后添加显式的块结束标记,并引入第二阶段BPE压缩以控制块粒度,该方法实现了高效、灵活且语言无关的表示。实证结果表明,该方法在性能上匹配或超越基于动态熵和空格的分块策略,同时保持紧凑词汇表。

原文摘要 · Abstract (English)

Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. However, they suffer from inefficiencies in representing rare words and require large embedding matrices. Character-level models address these issues but introduce performance bottlenecks, particularly in Transformer-based architectures. Recent hierarchical models attempt to merge the benefits of both paradigms by grouping characters into patches, but existing patching strategies either rely on whitespace-limiting applicability to certain languages, or require auxiliary models that introduce new dependencies. In this paper, we propose a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models. By appending explicit end-of-patch markers to BPE tokens and introducing a second-level BPE compression stage to control patch granularity, our method offers efficient, flexible, and language-agnostic representations. Empirical results demonstrate that our approach matches or exceeds the performance of dynamic entropy- and whitespace-based patching strategies, while maintaining a compact vocabulary.

分词BPE语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。