arXiv:2608.18025cs.LGcs.AI2026-08

提出词元化边界框架,指导音乐建模中何时切分、何时保留关系。

How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness

  • 以预测码长为标准,定义词元化起止边界:事实与状态分离。
  • 显式时间信息可降低预测码长,提升音高和时长预测精度。
  • 过度固定音程关系反而增加负担,可逆BPE压缩不保证性能提升。

GPT类模型凭借有限词汇量的可复用词元取得巨大成功,使词元接口成为现代序列建模的核心。符号音乐天然契合此范式:由离散音符事件和重复结构(如和弦、动机、乐句)组成。然而,当词元化扩展至语言之外,接口需针对各领域设计。现有方法虽有效,但缺乏统一标准判断词元应表征什么以及延伸多远。本文以预测码长为统一标准,提出有效-无损框架,界定词元化的起点与终点。事实-词元边界标志着观测决定的结构应进入词元接口,通过坐标构建等操作实现;在此接口内,载体可逆编码而不改变所表事实。词元-状态边界标志着词元化应停止:依赖上下文的关系应保留在模型状态计算中,而非预先固化于分词器。通过受控多种子符号音乐实验及独立语料库的时间干预复现验证,显式音乐时间始终降低预测码长,并提升音高与持续时间预测表现;调性框架规范化与音高因子分解进一步带来增益。固定五度圈音高坐标则增加预测码长,表明在上下文前强加固定音程关系会加重预测负担。可逆BPE虽缩短载体,但在所有种子下均增加预测码长,说明载体压缩本身不保证预测优势。

原文摘要 · Abstract (English)

GPT-style models have achieved remarkable success with finite vocabularies of reusable tokens, making the token interface a central component of modern sequence modeling. Symbolic music appears naturally compatible with this paradigm: it consists of discrete note events and recurring structures such as chords, motifs, and phrases. However, when tokenization moves beyond language, the interface must be specified for each domain. Existing work offers many effective designs, but no unified criterion for deciding what tokenization should represent and how far it should go. Using predictive codelength as a common criterion, we formulate the Effectiveness--Losslessness Framework to define where tokenization should begin and where it should end. The Fact--Token Boundary marks where observation-determined structure should enter the token interface, through operations such as coordinate construction. Within this interface, the resulting carrier may be reversibly recoded without changing the represented facts. The Token--State Boundary marks where tokenization should stop: relations that depend on context should remain for model-state computation rather than being fixed in advance by the tokenizer. We validate the framework through controlled multi-seed symbolic-music experiments, with an independent-corpus replication of the temporal intervention. Making musical time explicit consistently reduces predictive code and also improves pitch and duration prediction, while tonal-frame canonicalization and pitch factorization provide further gains. Fixed circle-of-fifths pitch coordinates instead increase predictive code, suggesting that imposing a fixed pitch relation before context can burden prediction. Reversible BPE substantially shortens the carrier but increases predictive codelength in every seed, showing that carrier compaction alone does not guarantee predictive gain.

词元化音乐建模无损编码序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。