合成数据+少量真实数据,提升水果检测模型在真实设备上的表现
Sim-to-Real Fruit Detection Using Synthetic Data: Quantitative Evaluation and Embedded Deployment with Isaac Sim
- 用Isaac Sim生成合成数据,与真实图像混合训练YOLO模型
- 混合训练比纯合成数据提升性能,接近纯真实数据效果
- 模型成功部署在Jetson Orin NX上,实现实时推理
本研究探讨在数据受限和嵌入式部署需求下,合成数据在模拟到真实迁移中的有效性。在NVIDIA Isaac Sim中生成合成数据集,并与有限的真实水果图像结合,分别在仅真实、仅合成和混合三种训练策略下训练基于YOLO的检测模型。在两个测试数据集上评估性能:一个为同域数据(训练条件匹配),另一个为域偏移数据(真实水果与不同背景)。结果表明,仅使用真实数据的模型准确率最高,纯合成数据模型因域差距表现较差。混合训练显著优于纯合成方案,性能接近真实数据训练,同时减少人工标注需求。在域偏移条件下,所有模型性能下降,但混合模型更具鲁棒性。训练模型经TensorRT优化后成功部署于Jetson Orin NX,实现实时推理。研究强调合成数据需与真实数据结合使用,且部署约束必须与检测精度一并考虑。
原文摘要 · Abstract (English)
This study investigates the effectiveness of synthetic data for sim-to-real transfer in object detection under constrained data conditions and embedded deployment requirements. Synthetic datasets were generated in NVIDIA Isaac Sim and combined with limited real-world fruit images to train YOLO-based detection models under real-only, synthetic-only, and hybrid regimes. Performance was evaluated on two test datasets: an in-domain dataset with conditions matching the training data and a domain shift dataset containing real fruit and different background conditions. Results show that models trained exclusively on real data achieve the highest accuracy, while synthetic-only models exhibit reduced performance due to a domain gap. Hybrid training strategies significantly improve performance compared to synthetic-only approaches and achieve results close to real-only training while reducing the need for manual annotation. Under domain shift conditions, all models show performance degradation, with hybrid models providing improved robustness. The trained models were successfully deployed on a Jetson Orin NX using TensorRT optimization, achieving real-time inference performance. The findings highlight that synthetic data is most effective when used in combination with real data and that deployment constraints must be considered alongside detection accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。