用预训练分割模型自动发现多动物交互中的关键点，省去人工标注。
Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision
- 利用预训练视频分割模型引导多主体关键点发现。
- 在果蝇、小鼠、大鼠视频中实现更优的关键点定位与行为分类。
- 可推广至蚂蚁、蜜蜂、人类等物种，适合跨物种行为分析。
通过多主体视频分析研究社会互动与集体行为在生物学中至关重要。尽管自监督关键点发现已成为减少人工标注需求的有前景方案，但现有方法在处理多个相互作用的同种、同色个体时仍表现不佳。为此，我们提出 B-KinD-multi，一种新方法，利用预训练视频分割模型引导多主体场景下的关键点发现，避免了在新实验设置和生物体上进行耗时的人工标注。大量评估表明，该方法在果蝇、小鼠和大鼠视频中显著提升了关键点回归精度与下游行为分类性能。此外，该方法对其他物种（包括蚂蚁、蜜蜂和人类）也具有良好的泛化能力，展示了其在多主体行为分析自动关键点标注中的广泛潜力。代码已公开：https://danielpkhalil.github.io/B-KinD-Multi
原文摘要 · Abstract (English)
The study of social interactions and collective behaviors through multi-agent video analysis is crucial in biology. While self-supervised keypoint discovery has emerged as a promising solution to reduce the need for manual keypoint annotations, existing methods often struggle with videos containing multiple interacting agents, especially those of the same species and color. To address this, we introduce B-KinD-multi, a novel approach that leverages pre-trained video segmentation models to guide keypoint discovery in multi-agent scenarios. This eliminates the need for time-consuming manual annotations on new experimental settings and organisms. Extensive evaluations demonstrate improved keypoint regression and downstream behavioral classification in videos of flies, mice, and rats. Furthermore, our method generalizes well to other species, including ants, bees, and humans, highlighting its potential for broad applications in automated keypoint annotation for multi-agent behavior analysis. Code available under: https://danielpkhalil.github.io/B-KinD-Multi
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。