小模型精调识别地中海饮食，准确率超大厂闭源模型。
OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet
- 用LoRA微调30亿参数视觉语言模型，专注地中海菜系识别。
- 食物识别准确率达92.96%，比最强基线高7.31%，比闭源模型高46%以上。
- 适合健康研究、饮食分析场景，开源可直接使用。
基于图像的饮食评估为自述食谱提供了可扩展的替代方案，但因类内差异大、菜品外观相似，细粒度食物识别仍具挑战。本研究提出OliveGemma，一个用于识别和推理地中海及欧洲菜系的视觉语言模型。该模型基于开源的PaliGemma-2-3B架构，通过LoRA在三个欧洲研究项目数据集（MedGR、ODIN、VIPPSTAR）合并的17,340张图像上进行微调，构建包含216个复合菜品类别的词汇表，并配以102,642条指令式问答数据，涵盖菜品识别、可能/可见食材、类别边界区分、视觉证据与整体食物理解。在三折交叉验证下，OliveGemma达到92.96%±0.91%的Top-1准确率，较最强CNN基线DenseNet-121高出7.31%，优于零样本前沿模型如Gemini Flash 3、Gemini 3.5、GPT-5.4 Mini和Claude Haiku 4.6，分别高出8%、46%和64%。此外，其在可能食材识别上的Exact-Set准确率为90.79%±1.3%。结果表明，小规模视觉语言模型经PEFT适配后，可在特定领域显著超越更大规模的闭源模型。模型已公开于Hugging Face，实验代码见GitHub。
原文摘要 · Abstract (English)
Image based dietary assessment offers a scalable alternative to self reported food diaries, yet fine-grained food recognition remains challenging due to high intra-class variability and visually similar dishes. This study presents OliveGemma, a vision language model for recognising and reasoning about Mediterranean and European cuisine. Built on the open-weight PaliGemma-2-3B architecture, OliveGemma is fine-tuned with LoRA on a unified corpus of 17,340 images from three European research project datasets (MedGR, ODIN, and VIPPSTAR), reconciled into a vocabulary of 216 composed dish categories and paired with 102,642 instruction style question-answer items covering dish recognition, likely and visible ingredients, class boundary discrimination, visual evidence and overall visual food understanding. Under a 3-fold cross-validation scheme, OliveGemma achieves a top-1 accuracy of 92.96% +/- 0.91%, exceeding the strongest CNN baseline (DenseNet-121) by 7.31% and outperforming zero-shot frontier models with exact instructions and bounded classes including Gemini Flash 3 and 3.5, GPT-5.4 Mini, and Claude Haiku 4.6 by 8%, 46%, and 64% respectively. Furthermore, OliveGemma demonstrates competitive performance on Top-3 and Top-5 accuracy, being second best across CNNs and frontier models, surpassed only by DenseNet-121. In addition, OliveGemma achieves 90.79% +/- 1.3% Exact-Set on the likely ingredients of the food categories. These results demonstrate that PEFT adaptation of a small VLM can surpass substantially larger proprietary models on specialised food recognition. The model is publicly available at https://huggingface.co/JamesZar/OliveGemma-3B and the experiments and results can be found at https://github.com/tsiokris/OliveGemma.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。