融合视觉与食材信息，提升快餐营养估算准确率。
Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion
- 设计视觉-食材特征融合框架，利用双模态信息增强预测。
- 在FastFood和Nutrition5k数据集上，营养估计精度显著提升。
- 适合对饮食健康、智能营养分析感兴趣的开发者与研究者。
营养估算对促进健康饮食、降低饮食相关健康风险具有重要意义。尽管食物分类和食材识别任务已取得进展，但因缺乏带营养标注的数据集，营养估算进展受限。为此，我们构建了包含84,446张图像、908类快餐的FastFood数据集，提供食材与营养标注。同时提出模型无关的视觉-食材特征融合（VIF²）方法，通过训练阶段的同义词替换与重采样策略提升食材鲁棒性，并采用食材感知的视觉特征融合模块结合视觉与食材特征实现精准营养预测。测试时利用大模型进行数据增强与多数投票，优化食材预测结果。在FastFood和Nutrition5k数据集上，基于ResNet、InceptionV3与ViT等不同骨干网络的实验验证了该方法有效性，证明了食材信息在营养估算中的关键作用。
原文摘要 · Abstract (English)
Nutrition estimation is an important component of promoting healthy eating and mitigating diet-related health risks. Despite advances in tasks such as food classification and ingredient recognition, progress in nutrition estimation is limited due to the lack of datasets with nutritional annotations. To address this issue, we introduce FastFood, a dataset with 84,446 images across 908 fast food categories, featuring ingredient and nutritional annotations. In addition, we propose a new model-agnostic Visual-Ingredient Feature Fusion (VIF$^2$) method to enhance nutrition estimation by integrating visual and ingredient features. Ingredient robustness is improved through synonym replacement and resampling strategies during training. The ingredient-aware visual feature fusion module combines ingredient features and visual representation to achieve accurate nutritional prediction. During testing, ingredient predictions are refined using large multimodal models by data augmentation and majority voting. Our experiments on both FastFood and Nutrition5k datasets validate the effectiveness of our proposed method built in different backbones (e.g., Resnet, InceptionV3 and ViT), which demonstrates the importance of ingredient information in nutrition estimation. https://huiyanqi.github.io/fastfood-nutrition-estimation/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。