用扩散模型生成罕见手术视频，缓解数据不平衡问题。
Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models
- 分两阶段生成：先用2D扩散模型建模空间内容，再加时间注意力保持连贯性。
- 通过拒绝采样筛选优质合成样本，提升数据集质量，使模型性能显著提升。
- 适合需要增强小样本类别的医疗视频分析研究者使用。
计算机辅助干预可提升术中引导效果，尤其依赖利用手术视频时空信息的深度学习方法。但手术视频数据集中常存在严重数据不平衡，制约高性能模型的发展。本文提出一种基于文本条件的两阶段扩散生成方法，用于合成罕见类别的手术视频。该方法以文本提示为条件，将空间与时间建模解耦：先使用2D潜在扩散模型捕捉空间内容，再引入时间注意力层确保时序一致性。此外，设计拒绝采样策略筛选最优合成样本，有效扩充数据集以缓解类别不平衡。在手术动作识别和术中事件预测两个下游任务上验证，加入合成视频后模型性能明显提升。代码已开源：https://gitlab.com/nct_tso_public/surgvgen。
原文摘要 · Abstract (English)
Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at https://gitlab.com/nct_tso_public/surgvgen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。