arXiv:2605.30080cs.CL2026-05

提出动态分块机制,提升无分词模型的压缩效率与训练稳定性。

Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model

论文配图:Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model
图 1 · 摘自论文原文
  • 基于课程学习动态调整分块压缩比,逐步优化压缩效率。
  • 在FineWeb-Edu 100B上达到与传统模型相当的每字节比特数(BPB)。
  • 适合追求高效、鲁棒的低级文本建模研究者使用。

无分词的层次化模型正成为大型语言模型的有力替代方案,可解决词汇设计复杂、未登录词错误及语言依赖等预处理问题。然而,这类字节级方法面临压缩比优化难题,直接影响模型对字节数据的处理性能。本文提出自适应目标动态分块(ATDC),一种新型字节压缩控制机制,用于增强层次架构中动态分块的效果。该方法采用课程学习,在训练过程中从低到高逐步调整压缩比,以稳定学习过程。我们分析了目标压缩比与最小内层块字节数(BPIC)的关系,可追踪训练中分块大小的变化。在FineWeb-Edu 100B数据集上的评估表明,配备ATDC的层次模型在比特/字节(BPB)表现上与传统字节级和词元级基线相当。此外,相比固定压缩比模型,该方法展现出更稳定的训练动态和更优的下游任务性能,同时保持字节级处理的固有鲁棒性与灵活性。

原文摘要 · Abstract (English)

Tokenization-free hierarchical models are emerging as a promising alternative to traditional Large Language Models (LLMs), addressing inherent preprocessing issues such as vocabulary design complexity, out-of-vocabulary (OOV) errors, and language-specific constraints. However, a significant challenge in these byte-level methods is the optimization of the compression ratio, a critical factor that dictates model performance for processing bytes data via chunks. In this paper, we propose Adaptive Targeted Dynamic Chunking (ATDC), a novel byte-compression control mechanism designed to enhance the effectiveness of dynamic chunking within hierarchical architectures. Our approach utilizes curriculum learning to progressively adjust the compression ratio during training, transitioning from low to high compression to stabilize the learning process. We provide an analysis establishing the relationship between the target compression ratio and Bytes-Per-Innermost-Chunk (BPIC), allowing for tracking of chunk-size evolution throughout the training phase. Evaluations conducted on the FineWeb-Edu 100B dataset demonstrate that hierarchical models equipped with ATDC achieve competitive Bits-Per-Byte (BPB) performance compared to conventional baselines operating at both byte and token levels. Furthermore, the proposed method exhibits more stable training dynamics and superior final performance across diverse downstream tasks compared to models using fixed compression ratios, while maintaining the inherent robustness and flexibility of byte-level processing.

层次模型字节级建模动态分块压缩比优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。