arXiv:2511.00793cs.MMcs.SD2025-11

用手势实时生成连续音乐,延迟仅30毫秒。

Gesture2Music: A Low-Latency Real-Time Framework for Continuous Gesture-Driven Music Generation

  • 用因果卷积网络从手势序列直接预测音乐参数。
  • 在自建数据集上实现30毫秒延迟,音乐连续性显著提升。
  • 适合音乐交互、人机协同创作等实时场景。

手势驱动的音乐生成是无需接触且富有表现力的新型人机交互范式。现有方法多将任务视为孤立的手势分类,或先映射到MIDI等符号输出再单独渲染,限制了时间连续性和实时响应。本文提出Gesture2Music,一种低延迟流式框架,可从实时摄像头输入中连续生成音乐。系统处理身体与手部关键点序列,利用因果时间卷积网络(TCN)预测音符级音乐控制事件,包括音高、八度、起始、持续、振幅和活动状态。由于现有手势-音符数据集通常仅有孤立单音记录,缺乏连续演奏序列,研究引入合成流生成策略,通过拼接单音片段并推导启发式时间事件标签构建连续手势流。进一步采用时间一致性与频谱代理损失,减少预测抖动并促进音频一致性。推理时,预测事件通过预设音符样本、节奏量化及音阶约束滤波器渲染为连续音乐。在包含21种手势-音符类别的自建数据集上,覆盖七个音高跨越三个音区,实验验证了稳定实时性能、30毫秒低延迟以及显著提升的时间连续性。

原文摘要 · Abstract (English)

Gesture-driven music generation is an emerging human-computer interaction paradigm for touch-free and expressive musical interaction. However, many existing approaches treat the task as isolated gesture classification or map gestures to symbolic outputs such as MIDI followed by a separate rendering stage, which limits temporal continuity and real-time responsiveness. This work presents Gesture2Music, a low-latency streaming framework for continuous gesture-driven music generation from live webcam feed. The system processes sequences of body and hand landmarks and uses a causal temporal convolutional network (TCN) to predict note-level musical control events, including pitch, octave, onset, sustain, amplitude, and activity state. Because available gesture-note datasets typically contain only isolated single-note recordings rather than continuous performance sequences, a synthetic stream generation strategy is introduced to construct continuous gesture streams by concatenating single-note clips and deriving heuristic temporal event labels. Temporal consistency and spectral proxy losses are further used to reduce prediction jitter and encourage audio-consistent outputs. During inference, predicted musical events are rendered into continuous music using predefined note samples with rhythmic quantization and scale-constrained filtering for improved musical stability. Experiments on a custom gesture-to-music dataset with 21 gesture-note classes spanning seven tones across three pitch levels demonstrate stable real-time performance, low inference latency of 30\,ms, and improved temporal continuity.

手势生成实时音乐低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。