用生成式模型让灯光自动随音乐变化,更自然生动。
Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?
- 将灯光控制视为生成任务,用BART模型直接从音频生成灯光参数。
- 在自建数据集上,生成效果接近专业工程师水平,优于传统规则方法。
- 适合想做音乐-灯光联动研究或智能演出系统开发的人。
舞台灯光是现场音乐演出中至关重要的组成部分,影响着表演者与观众的沉浸体验。近年来,由于聘请或培训专业灯光师成本高昂,自动舞台灯光控制(ASLC)受到越来越多关注。然而,现有大多数ASLC方案仅将音乐分类为有限类别,并映射到预设灯光模式,导致结果单调且缺乏合理性。为此,本文提出Skip-BART,一个端到端的生成式模型,直接学习经验丰富的灯光师行为,以预测生动、类人的舞台灯光效果。据我们所知,这是首个将ASLC视为生成任务而非单纯分类问题的工作。该方法基于BART模型,以音频为输入,输出灯光色相与明度(强度),并引入新颖的跳跃连接机制,强化音乐与灯光在帧网格间的关联。为应对数据稀缺问题,我们构建了首个舞台灯光数据集,并采用多种预训练与迁移学习技术,提升小样本下的模型训练效果。通过定量分析与人工评估验证,结果显示Skip-BART在所有评价指标上均优于传统规则方法,与真实灯光师表现差距较小。为促进后续研究,我们已将自采集数据集、代码及训练好的模型参数公开于 https://github.com/RS2002/Skip-BART。
原文摘要 · Abstract (English)
Stage lighting is a vital component in live music performances, shaping an engaging experience for both musicians and audiences. In recent years, Automatic Stage Lighting Control (ASLC) has attracted growing interest due to the high costs of hiring or training professional lighting engineers. However, most existing ASLC solutions only classify music into limited categories and map them to predefined light patterns, resulting in formulaic and monotonous outcomes that lack rationality. To address this gap, this paper presents Skip-BART, an end-to-end model that directly learns from experienced lighting engineers and predict vivid, human-like stage lighting. To the best of our knowledge, this is the first work to conceptualize ASLC as a generative task rather than merely a classification problem. Our method adapts the BART model to take audio music as input and produce light hue and value (intensity) as output, incorporating a novel skip connection mechanism to enhance the relationship between music and light within the frame grid. To address the lack of available datasets, we create the first stage lighting dataset, along with several pre-training and transfer learning techniques to improve model training with limited data. We validate our method through both quantitative analysis and an human evaluation, demonstrating that Skip-BART outperforms conventional rule-based methods across all evaluation metrics and shows only a limited gap compared to real lighting engineers. To support further research, we have made our self-collected dataset, code, and trained model parameters available at https://github.com/RS2002/Skip-BART .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。