用图文生成增强数据,提升稀有鱼类识别准确率
FishAI 2.0: Marine Fish Image Classification with Multi-modal Few-shot Learning
- 结合大模型生成文本,通过扩散模型合成图像扩增数据
- 稀有类仅10样本下仍达91.67%分类准确率
- 适合海洋生态监测中样本稀缺的物种识别
传统海洋生物图像识别面临数据不全与模型精度不足的问题,尤其在罕见物种的少样本条件下,数据稀缺严重制约性能。为此,本文提出FishAI 2.0框架,融合多模态少样本深度学习与图像生成技术进行数据增强。首先,使用层次化海洋鱼类基准数据集训练模型;针对稀有类别数据不足问题,利用大语言模型DeepSeek生成高质量文本描述,并输入Stable Diffusion 2,通过分层扩散策略提取隐空间编码,构建视觉-文本联合特征空间。增强后的图文数据输入基于对比语言-图像预训练(CLIP)的模型,实现鲁棒的少样本图像识别。实验表明,鱼科级别下Top-1准确率达91.67%,Top-5达97.97%,显著优于基线CLIP与ViT模型,尤其在少于10样本的少数类上表现突出。属和种级别分别达到87.58%与85.42%的Top-1准确率,具备实际应用价值。整体提升了海洋鱼类识别的效率与准确性,为海洋生态监测与保护提供可扩展的技术方案。
原文摘要 · Abstract (English)
Traditional marine biological image recognition faces challenges of incomplete datasets and unsatisfactory model accuracy, particularly for few-shot conditions of rare species where data scarcity significantly hampers the performance. To address these issues, this study proposes an intelligent marine fish recognition framework, FishAI 2.0, integrating multimodal few-shot deep learning techniques with image generation for data augmentation. First, a hierarchical marine fish benchmark dataset, which provides a comprehensive data foundation for subsequent model training, is utilized to train the FishAI 2.0 model. To address the data scarcity of rare classes, the large language model DeepSeek was employed to generate high-quality textual descriptions, which are input into Stable Diffusion 2 for image augmentation through a hierarchical diffusion strategy that extracts latent encoding to construct a multimodal feature space. The enhanced visual-textual datasets were then fed into a Contrastive Language-Image Pre-Training (CLIP) based model, enabling robust few-shot image recognition. Experimental results demonstrate that FishAI 2.0 achieves a Top-1 accuracy of 91.67 percent and Top-5 accuracy of 97.97 percent at the family level, outperforming baseline CLIP and ViT models with a substantial margin for the minority classes with fewer than 10 training samples. To better apply FishAI 2.0 to real-world scenarios, at the genus and species level, FishAI 2.0 respectively achieves a Top-1 accuracy of 87.58 percent and 85.42 percent, demonstrating practical utility. In summary, FishAI 2.0 improves the efficiency and accuracy of marine fish identification and provides a scalable technical solution for marine ecological monitoring and conservation, highlighting its scientific value and practical applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。