提出分层动态分块模型，让无分词器语言模型更好处理形态丰富的语言。
H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages
- 通过层次化动态分块学习语言结构，无需人工分词。
- 在波斯语上实现0.159的比特每字符，比基线提升12%压缩率。
- 适合研究形态丰富语言或无分词器模型的学者使用。
字节级语言模型虽避免了脆弱的分词器，但在形态丰富的语言（如波斯语）中面临计算挑战，因单词常占多个字节。本文提出H-NET++，一种通过端到端训练学习语言学启发式分段的分层动态分块模型。关键创新包括：(1) 轻量级Transformer上下文混合器（190万参数），支持跨块注意力；(2) 两级潜在超先验，保证文档级一致性；(3) 对拼写异常（如波斯语中的零宽非连接符）的专门处理；(4) 分阶段序列长度的课程训练。在14亿字节的波斯语语料上，H-NET++达到最先进水平：相比基于BPE的GPT-2-fa，比特每字符（BPB）降低0.159（压缩率提升12%），在ParsGLUE上提升5.4个百分点，对零宽非连接符扰动的鲁棒性提高53%，在真实形态边界上的F1达73.8%。所学分块与波斯语形态高度一致，且无需显式监督，证明分层动态分块是形态丰富语言的有效无分词器解决方案，同时保持计算高效。
原文摘要 · Abstract (English)
Byte-level language models eliminate fragile tokenizers but face computational challenges in morphologically-rich languages (MRLs), where words span many bytes. We propose H-NET++, a hierarchical dynamic-chunking model that learns linguistically-informed segmentation through end-to-end training. Key innovations include: (1) a lightweight Transformer context-mixer (1.9M parameters) for cross-chunk attention, (2) a two-level latent hyper-prior for document-level consistency, (3) specialized handling of orthographic artifacts (e.g. Persian ZWNJ), and (4) curriculum-based training with staged sequence lengths. On a 1.4B-token Persian corpus, H-NET++ achieves state-of-the-art results: 0.159 BPB reduction versus BPE-based GPT-2-fa (12% better compression), 5.4pp gain on ParsGLUE, 53% improved robustness to ZWNJ corruption, and 73.8% F1 on gold morphological boundaries. Our learned chunks align with Persian morphology without explicit supervision, demonstrating that hierarchical dynamic chunking provides an effective tokenizer-free solution for MRLs while maintaining computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。