构建了包含语言、动作和笑声的喜剧表演多模态数据库,用于分析喜剧节奏。
Timing In stand-up Comedy: Text, Audio, Laughter, Kinesics (TIC-TALK): Pipeline and Database for the Multimodal Study of Comedic Timing
- 用BERTopic与Whisper-AT等模型实现多模态数据自动对齐
- 发现表演者肢体动能越强,观众笑得越少(相关系数-0.75)
- 适合研究喜剧表演、人机交互或情感计算的学者使用
单口喜剧常被从语言内容角度研究,但现场演出同样依赖身体表现与观众反应。我们提出TIC-TALK,一个包含5400多个时间对齐话题片段的多模态资源,覆盖90部专业拍摄的单口喜剧特辑(2015–2024)。该流程结合BERTopic进行60秒主题分割,使用密集句向量;Whisper-AT检测0.8秒级笑声;微调YOLOv8-cls识别镜头类型;YOLOv8s-pose以1帧/秒提取原始17关节骨骼坐标,不预先聚类,从而计算连续运动信号——如手臂展开度、动能和躯干倾斜度,作为表演动态代理。所有信号通过分层时间包含对齐,无需重采样,每个话题段存储sentence-BERT嵌入,支持后续相似性与聚类任务。作为具体应用,我们研究24个主题下的笑声动态:动能与观众笑声率负相关(r = -0.75,N=24),符合“静止—笑点”模式;个人与身体主题引发更多笑声,而政治主题较少;特写镜头比例与笑声正相关(r = +0.28),符合反应蒙太奇特征。
原文摘要 · Abstract (English)
Stand-up comedy, and humor in general, are often studied through their verbal content. Yet live performance relies just as much on embodied presence and audience feedback. We introduce TIC-TALK, a multimodal resource with 5,400+ temporally aligned topic segments capturing language, gesture, and audience response across 90 professionally filmed stand-up comedy specials (2015-2024). The pipeline combines BERTopic for 60 s thematic segmentation with dense sentence embeddings, Whisper-AT for 0.8 s laughter detection, a fine-tuned YOLOv8-cls shot classifier, and YOLOv8s-pose for raw keypoint extraction at 1 fps. Raw 17-joint skeletal coordinates are retained without prior clustering, enabling the computation of continuous kinematic signals-arm spread, kinetic energy, and trunk lean-that serve as proxies for performance dynamics. All streams are aligned by hierarchical temporal containment without resampling, and each topic segment stores its sentence-BERT embedding for downstream similarity and clustering tasks. As a concrete use case, we study laughter dynamics across 24 thematic topics: kinetic energy negatively predicts audience laughter rate (r = -0.75, N = 24), consistent with a stillness-before-punchline pattern; personal and bodily content elicits more laughter than geopolitical themes; and shot close-up proportion correlates positively with laughter (r = +0.28), consistent with reactive montage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。