用生成模型解决手术视频数据不平衡问题，提升罕见动作识别效果。
Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model
- 双预测扩散框架同时去噪图像和光流，增强运动建模能力。
- 生成数据使动作识别等任务性能提升10%-20%。
- 无需密集标注，仅需稀疏信号即可控制生成过程。
手术视频数据集对场景理解与术中辅助至关重要，但普遍存在数据不平衡问题，罕见动作与工具样本不足，影响下游模型鲁棒性。本文提出SurgiFlowVid，一种稀疏可控的视频扩散生成框架，用于生成稀缺类别的手术视频。该方法引入双预测扩散模块，联合去噪RGB帧与光流，提供时序归纳偏置，提升有限样本下的运动建模能力。同时，采用稀疏视觉编码器，基于轻量级信号（如稀疏分割掩码或RGB帧）控制生成过程，实现可控性且无需密集标注。在三个手术数据集上验证，涵盖动作识别、工具存在检测与腹腔镜运动预测任务。所生成合成数据相较基线模型带来10%-20%的性能提升，证明SurgiFlowVid是缓解数据不平衡、推动手术视频理解的有效策略。
原文摘要 · Abstract (English)
Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with $SurgiFlowVid$, a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of 10-20% over competitive baselines, establishing $SurgiFlowVid$ as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。