无需标注数据，实时精准估测音高，模型仅13万参数
PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective
- 采用孪生结构与移调等变目标，自监督学习音高分布
- 在音乐和语音数据集上超越自监督基线，媲美有监督方法
- 轻量低延迟，适合实时音频处理应用
本文提出PESTO，一种基于孪生架构的自监督单音高估测方法。模型处理可变$Q$变换（VQT）帧，预测音高分布。通过引入托普利茨全连接层，实现对音高的平移等变性。利用对VQT帧进行移位裁剪构造音高偏移对，训练时采用新型类别级移调等变目标，无需人工标注。该设计使模型仅需13万参数即达优异性能。在MIR-1K、MDB-stem-synth和PTDB等音乐与语音数据集上的评估表明，PESTO不仅优于现有自监督方法，还达到与有监督方法相当的水平，展现出卓越的跨数据集泛化能力。此外，通过使用缓存卷积实现可流式处理的VQT，结合模型低于10毫秒的延迟和极小参数量，使PESTO特别适用于实时应用场景。
原文摘要 · Abstract (English)
In this paper, we introduce PESTO, a self-supervised learning approach for single-pitch estimation using a Siamese architecture. Our model processes individual frames of a Variable-$Q$ Transform (VQT) and predicts pitch distributions. The neural network is designed to be equivariant to translations, notably thanks to a Toeplitz fully-connected layer. In addition, we construct pitch-shifted pairs by translating and cropping the VQT frames and train our model with a novel class-based transposition-equivariant objective, eliminating the need for annotated data. Thanks to this architecture and training objective, our model achieves remarkable performances while being very lightweight ($130$k parameters). Evaluations on music and speech datasets (MIR-1K, MDB-stem-synth, and PTDB) demonstrate that PESTO not only outperforms self-supervised baselines but also competes with supervised methods, exhibiting superior cross-dataset generalization. Finally, we enhance PESTO's practical utility by developing a streamable VQT implementation using cached convolutions. Combined with our model's low latency (less than 10 ms) and minimal parameter count, this makes PESTO particularly suitable for real-time applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。