用分层决策提升细粒度食物分析,精度显著优于现有模型
FoodCHA: Multi-Modal LLM Agent for Fine-Grained Food Analysis

- 分层推理:先类别后子类再烹饪方式,逐步锁定目标
- 子类识别精度提升38.2%,烹饪方式识别提升153.2%
- 轻量级模型部署便捷,适合移动端实时饮食监测
随带摄像头的移动设备和可穿戴设备普及,使餐食图像采集变得便捷,食物识别成为实时饮食监测的关键。然而,真实场景下的食物图像存在类内相似度高、单图含多种食物等问题。尽管深度学习在粗粒度分类上表现良好,但在捕捉烹饪方式等细粒度属性时仍存困难。现代视觉-语言模型的开放式生成常产生非标准标签,限制实际应用。本文提出FoodCHA,一种多模态智能体框架,将食物识别重构为分层决策过程:先利用高层类别引导子类识别,再以子类指导烹饪方式判断,增强语义一致性与属性区分能力。为保障可部署性,采用紧凑的Moondream-2B视觉语言模型,在保持强推理能力的同时降低计算与内存开销。在FoodNExTDB数据集上的实验表明,FoodCHA在类别识别精度上比Food-Llama-3.2-11B高出13.8%,子类识别提升38.2%,烹饪方式分类精度更是提升153.2%。
原文摘要 · Abstract (English)
The widespread adoption of camera-equipped mobile devices and wearables has enabled convenient capture of meal images, making food recognition a key component for real time dietary monitoring. However, real-world food images present challenges due to high intra-class similarity and the frequent presence of multiple food items within a single image. While deep learning models achieve strong performance in coarse grained classification, they often struggle to capture fine-grained attributes such as cooking style. Moreover, open-ended generation in modern vision-language models can produce non-canonical labels, limiting their practical deployment. We propose FoodCHA, a multimodal agentic framework that reformulates food recognition as a hierarchical decision-making process. By progressively anchoring predictions, FoodCHA guides subcategory identification using high-level categories and guides cooking style recognition using subcategories, improving semantic consistency and attribute-level discrimination. To ensure practical deployability, FoodCHA utilizes the compact Moondream-2B vision language model, which provides strong reasoning capability while maintaining lower computational and memory overhead. Experiments on FoodNExTDB show that FoodCHA outperforms Food-Llama-3.2-11B by 13.8% and 38.2% in category and subcategory recognition precision, respectively, and achieves a striking 153.2% improvement in cooking style classification precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。