用大模型生成食物文本描述，提升长尾食品数据的跨域识别效果。
LLMs-based Augmentation for Domain Adaptation in Long-tailed Food Datasets
- 通过大模型解析图像生成菜品名称和配料
- 在共享空间对齐图文特征，缓解跨域差异
- 适合处理长尾分布与细粒度分类的食品识别任务
基于互联网爬取的食品图像与用户实际拍摄的图像存在显著视觉差异，导致领域偏移问题；同时真实食品数据集通常呈长尾分布，且不同类别间细微差异使识别困难。本文提出一种基于大语言模型（LLMs）的框架，首先利用LLMs解析图像生成菜品名称和成分信息，再将生成的文本与来自不同领域的图像映射到共享嵌入空间以最大化模态间相似性，最后融合对齐后的多模态特征进行识别。该简单框架在两个食品数据集上均优于针对长尾分布、领域自适应和细粒度分类设计的现有方法。
原文摘要 · Abstract (English)
Training a model for food recognition is challenging because the training samples, which are typically crawled from the Internet, are visually different from the pictures captured by users in the free-living environment. In addition to this domain-shift problem, the real-world food datasets tend to be long-tailed distributed and some dishes of different categories exhibit subtle variations that are difficult to distinguish visually. In this paper, we present a framework empowered with large language models (LLMs) to address these challenges in food recognition. We first leverage LLMs to parse food images to generate food titles and ingredients. Then, we project the generated texts and food images from different domains to a shared embedding space to maximize the pair similarities. Finally, we take the aligned features of both modalities for recognition. With this simple framework, we show that our proposed approach can outperform the existing approaches tailored for long-tailed data distribution, domain adaptation, and fine-grained classification, respectively, on two food datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。