让模型更聪明地选择计算重点，自动聚焦难预测位置。
SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models
- 用新指标衡量分段质量，看起点是否落在难预测处。
- 在10亿参数规模下，提升准确率与效率平衡，计算更精准。
- 适合想优化序列建模效率的开发者和研究者。
层级序列模型通过学习分段替代固定分词，压缩长字节序列以实现高效自回归建模。尽管近期端到端方法仅靠语言建模目标即可学习有意义的边界，但难以定量评估并系统引导计算分配位置。本文提出一种无需依赖路由器的边界质量度量——边界增强度B，用于衡量分段起始点是否集中在高下一字节意外性（surprisal）的位置。基于此度量，我们提出Sombrero方法，通过置信度对齐边界损失将边界导向预测难度高的区域，并在输入层采用置信度加权平滑，而非对已生成块进行平滑，从而稳定边界学习。在涵盖英语、德语文本以及代码和数学内容的UTF-8语料库上，10亿参数规模下，Sombrero显著改善了准确率与效率的权衡，使计算更一致地对齐于难预测位置。
原文摘要 · Abstract (English)
Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end methods can learn meaningful boundaries from the language-modeling objective alone, it remains difficult to quantitatively assess and systematically steer where compute is spent. We introduce a router-agnostic metric of boundary quality, boundary enrichment B, which measures how strongly chunk starts concentrate on positions with high next-byte surprisal. Guided by this metric, we propose Sombrero, which steers boundary placement toward predictive difficulty via a confidence-alignment boundary loss and stabilizes boundary learning by applying confidence-weighted smoothing at the input level rather than on realized chunks. On 1B scale, across UTF-8 corpora covering English and German text as well as code and mathematical content, Sombrero improves the accuracy-efficiency trade-off and yields boundaries that more consistently align compute with hard-to-predict positions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。