提出自对齐框架PairAlign,用更短的符号序列保留语音结构信息。
PairAlign: A Framework for Autoregressive Tokenization via Self-Alignment with Applications to Audio Tokenization

- 通过自对齐训练,让模型生成紧凑且有序的语音符号序列
- 在TIMIT数据集上减少70.65%的令牌数,编辑距离降低91.87%
- 适合需要低速率、高结构保真的语音处理任务
现代学习系统用连续向量表示感知信号,但比较、检索、记忆、对齐和推理常依赖符号化操作。语言中已有分词作为接口,而语音与音频需学习该接口。现有音频分词器依赖局部量化、聚类或重构,导致序列一致性、紧凑性、长度、终止及编辑几何仅间接控制。我们提出PairAlign框架,通过自对齐实现紧凑音频分词。编码器将语音映射为连续条件,自回归解码器从起始符(BOS)到结束符(EOS)生成令牌。给定两个内容保持不变的视图,PairAlign推导出标准锚点目标,并训练两视图共同预测该目标,同时以同批次无关目标作为竞争序列。先学习从矢量量化(VQ)目标出发的自回归桥接,再过渡到带有接地和防坍缩控制的EMA教师自对齐。在3秒语音上,PairAlign生成更短且非退化的序列,同时保持有序配对视图一致性。在TIMIT检索归档上,运行速率达8.28令牌/秒,归档令牌数减少70.65%,编辑距离代理工作量降低91.87%相较于第一阶段。正负样本探测显示在音素不交、三元语法不交及时间重排负例间有强分离能力,而事后率控的BPE无法恢复相同紧凑性-一致性性能点。结果揭示速率-粒度权衡:PairAlign并非在所有局部指标上均优于更密集分词器,但提供更低速率的符号接口,保留有序与关系结构。概念上,PairAlign遵循JEPA式预测学习,预测抽象目标而非重建输入;此处目标为学习得到的可变长度符号序列。
原文摘要 · Abstract (English)
Modern learning systems represent perceptual signals with continuous vectors, but comparison, retrieval, memory, alignment, and reasoning are often symbolic. In language, tokens provide this interface; for speech and audio, it must be learned. Existing audio tokenizers rely on local quantization, clustering, or reconstruction, leaving sequence consistency, compactness, length, termination, and edit geometry only indirectly controlled. We introduce PairAlign, a framework for compact audio tokenization through autoregressive self-alignment. An encoder maps speech to a continuous condition, and an autoregressive decoder emits tokens from BOS to EOS. Given two content-preserving views, PairAlign derives a canonical anchor target and trains both views to predict it, with unrelated in-batch targets as competing sequences. It first learns an autoregressive bridge from VQ targets and then transitions to EMA-teacher self-alignment with grounding and anti-collapse controls. On 3 s speech, PairAlign produces shorter, non-degenerate sequences while retaining ordered paired-view consistency. On the TIMIT retrieval archive, it operates at 8.28 tokens/s, reducing archive token count by 70.65% and the edit-distance work proxy by 91.87% relative to Stage I. Positive-negative probes show strong separation across phone-disjoint, trigram-disjoint, and temporally rearranged negatives, while rate-controlled post-hoc BPE does not recover the same compactness-consistency operating point. These results expose a rate-granularity trade-off: PairAlign does not uniformly outperform denser tokenizers on every local metric, but provides a lower-rate symbolic interface preserving ordered and relational structure. Conceptually, PairAlign follows JEPA-style predictive learning by predicting an abstract target rather than reconstructing the input; here, the target is a learned variable-length symbolic sequence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。