将录音直接转为可玩的节奏游戏谱，无需人工标注
STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts
- 分阶段混合模型：音符检测+乐器识别+声学对齐，端到端生成游戏谱
- 鼓类音符检测准确率达83.8%，其他乐器在±100毫秒内表现良好
- 适用于音乐游戏制作者、开源社区及音频转谱研究者
我们提出STRUM（频谱音符转录与节奏理解模型），一种将原始录音直接转换为可玩的Clone Hero/YARG游戏谱（支持鼓、吉他、贝斯、人声、键盘）的端到端音频转谱管道，无需任何先验标注信息。STRUM采用多阶段混合架构：鼓类使用双阶段CRNN节拍检测器与六模型集成分类器；吉他与贝斯采用神经节拍检测结合单音调追踪；人声采用词对齐的语音识别；键盘则通过频谱检测实现。在基于单一音频质量标准（htdemucs_6s源分离后1秒鼓音轨均方根值）筛选的30首歌曲基准上评估，鼓类音符检测F1达到0.838，贝斯为0.694，吉他为0.651，人声为0.539（容忍误差±100毫秒，每曲全局偏移搜索）。报告了七个鼓处理组件的完整消融实验（配对威尔科克森检验）、社区谱面中真实时间与音频时间分布分析，以及鼓分类器的逐类混淆矩阵。代码、模型权重和完整基准清单已公开。
原文摘要 · Abstract (English)
We present STRUM (Spectral Transcription and Rhythm Understanding Model), an audio-to-chart pipeline that converts raw recordings into playable Clone Hero / YARG charts for drums, guitar, bass, vocals, and keys without any oracle metadata. STRUM is a multi-stage hybrid: a two-stage CRNN onset detector and a six-model ensemble classifier for drums; neural onset detectors with monophonic pitch tracking for guitar and bass; word-aligned ASR for vocals; and spectral keyboard detection for keys. We evaluate on a 30-song in-envelope benchmark constructed by screening candidate songs on a single audio-quality criterion -- the median 1-second drum-stem RMS after htdemucs_6s source separation. On this benchmark STRUM achieves drums onset F1 = 0.838, bass F1 = 0.694, guitar F1 = 0.651, and vocals F1 = 0.539 at a +/- 100 ms tolerance with per-song global offset search. We report a complete ablation of seven drum-pipeline components with paired per-song Wilcoxon tests, an analysis of ground-truth-to-audio timing distributions in community Clone Hero charts, and a per-class confusion matrix for the drum classifier. Code, model weights, and the full benchmark manifest are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。