统一转录歌词与音符并自动对齐,无需预处理
SongTrans: An unified song transcription and alignment method for lyrics and notes
- 用自回归与非自回归模块联合预测歌词、音符和时长
- 在歌词与音符转录任务上达到最新水平,首次实现自动对齐
- 支持带伴奏歌曲,适合实际音乐生成应用
歌唱语音合成的发展高度依赖高质量标注数据。现有工具虽可分别完成歌词或音符转录,但均需耗时的预处理(如人声与伴奏分离),且多为单一任务设计,难以实现歌词与音符的精确对齐。为此,我们首先通过优化现有工具并人工标注大量歌词-音符对构建数据集;随后基于该数据训练了统一的SongTrans模型,可直接从音频中同时转录歌词、音符并完成对齐,无需任何预处理。该模型包含两个模块:(1) 自回归模块预测歌词及每个词对应的时长和音符编号;(2) 非自回归模块预测音符的音高与时长。实验表明,SongTrans在歌词与音符转录任务上均达到当前最优性能,并首次实现歌词-音符自动对齐。结果还显示其能有效适应不同风格歌曲(包括带伴奏歌曲),具备良好的实用性。
原文摘要 · Abstract (English)
The quantity of processed data is crucial for advancing the field of singing voice synthesis. While there are tools available for lyric or note transcription tasks, they all need pre-processed data which is relatively time-consuming (e.g., vocal and accompaniment separation). Besides, most of these tools are designed to address a single task and struggle with aligning lyrics and notes (i.e., identifying the corresponding notes of each word in lyrics). To address those challenges, we first design a pipeline by optimizing existing tools and annotating numerous lyric-note pairs of songs. Then, based on the annotated data, we train a unified SongTrans model that can directly transcribe lyrics and notes while aligning them simultaneously, without requiring pre-processing songs. Our SongTrans model consists of two modules: (1) the \textbf{Autoregressive module} predicts the lyrics, along with the duration and note number corresponding to each word in a lyric. (2) the \textbf{Non-autoregressive module} predicts the pitch and duration of the notes. Our experiments demonstrate that SongTrans achieves state-of-the-art (SOTA) results in both lyric and note transcription tasks. Furthermore, it is the first model capable of aligning lyrics with notes. Experimental results demonstrate that the SongTrans model can effectively adapt to different types of songs (e.g., songs with accompaniment), showcasing its versatility for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。