用视觉变压器生成提示，让SAM自动分割食物图像，省去繁琐标注。
Weakly Supervised Food Image Segmentation using Vision Transformers and Segment Anything Model
- 用ViT的激活图生成SAM提示，实现弱监督分割。
- 在FoodSeg103上达到0.54 mIoU，每图生成2.4个食物掩码。
- 适合需要快速标注或营养追踪的应用场景。
本文提出一种基于视觉变换器（ViTs）与通用分割模型（SAM）的弱监督食物图像语义分割方法。利用Swin Transformer生成类激活图（CAMs）作为SAM的输入提示，实现无需像素级标注的训练。该方法在FoodSeg103数据集上平均生成2.4个非背景掩码，多掩码场景下获得0.54 mIoU。通过结合图像预处理和单/多掩码生成策略，进一步提升分割质量。该方法可加速食物图像标注流程，或集成于营养追踪系统中。
原文摘要 · Abstract (English)
In this paper, we propose a weakly supervised semantic segmentation approach for food images which takes advantage of the zero-shot capabilities and promptability of the Segment Anything Model (SAM) along with the attention mechanisms of Vision Transformers (ViTs). Specifically, we use class activation maps (CAMs) from ViTs to generate prompts for SAM, resulting in masks suitable for food image segmentation. The ViT model, a Swin Transformer, is trained exclusively using image-level annotations, eliminating the need for pixel-level annotations during training. Additionally, to enhance the quality of the SAM-generated masks, we examine the use of image preprocessing techniques in combination with single-mask and multi-mask SAM generation strategies. The methodology is evaluated on the FoodSeg103 dataset, generating an average of 2.4 masks per image (excluding background), and achieving an mIoU of 0.54 for the multi-mask scenario. We envision the proposed approach as a tool to accelerate food image annotation tasks or as an integrated component in food and nutrition tracking applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。