用生成视频动态物体增强数据,提升视频实例分割效果
SDI-Paste: Synthetic Dynamic Instance Copy-Paste for Video Instance Segmentation
- 用生成模型合成随时间变化的动态物体,自动复制粘贴到目标视频中
- 在Youtube-VIS 2021上实现+2.9 AP(6.5%)和+2.1 AP(4.9%)提升
- 适合做视频分割、目标跟踪的研究者使用,尤其关注数据增强
Copy-Paste等数据增强方法被证明是低成本扩展训练数据的有效手段。尽管已在图像任务中广泛应用,但针对视频任务的可扩展实现仍缺乏。本文利用生成模型在视频保真度上的进步,探索将合成物体引入现有视频数据集以人工扩充实例池的方法。我们获取了包含随时间动态变化物体的合成视频序列,通过精心设计的流水线自动分割并将其复制粘贴至任意目标背景视频的帧中。该方法称为合成动态实例复制粘贴(SDI-Paste),在视频实例分割任务上进行测试,该任务需同时完成检测、分割与跨帧追踪。在主流的Youtube-VIS 2021数据集上,基于两个不同主流网络作为基线,分别获得+2.9 AP(6.5%)和+2.1 AP(4.9%)的显著提升。代码与模型已公开。
原文摘要 · Abstract (English)
Data augmentation methods such as Copy-Paste have been studied as effective ways to expand training datasets while incurring minimal costs. While such methods have been extensively implemented for image level tasks, we found no scalable implementation of Copy-Paste built specifically for video tasks. In this paper, we leverage the recent growth in video fidelity of generative models to explore effective ways of incorporating synthetically generated objects into existing video datasets to artificially expand object instance pools. We first procure synthetic video sequences featuring objects that morph dynamically with time. Our carefully devised pipeline automatically segments then copy-pastes these dynamic instances across the frames of any target background video sequence. We name our video data augmentation pipeline Synthetic Dynamic Instance Copy-Paste, and test it on the complex task of Video Instance Segmentation which combines detection, segmentation and tracking of object instances across a video sequence. Extensive experiments on the popular Youtube-VIS 2021 dataset using two separate popular networks as baselines achieve strong gains of +2.9 AP (6.5%) and +2.1 AP (4.9%). We make our code and models publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。