arXiv:2508.17037cs.CV2025-08

通过融合视觉与文本特征,提升食物图文检索精度。

F4-ITS: Fine-grained Feature Fusion for Food Image-Text Search

  • 用视觉-语言模型生成文本描述,增强图像查询表达能力
  • 基于食材预测重排结果,顶k检索准确率提升28.6%
  • 小模型加文本融合后可超越大模型,适合资源受限场景

数字食物内容的激增催生了对细粒度视觉理解与检索系统的需求。本文针对食物图像到文本匹配这一挑战性任务,提出F4-ITS:一种无需训练、由视觉语言模型引导的细粒度特征融合框架,显著提升多模态特征表示能力。方法包含两项关键贡献:(1) 单向与双向多模态融合策略,将图像嵌入与VLM生成的文本描述结合,增强查询表达力;(2) 基于食材预测的新型重排序机制,用于改进top-k检索结果,提升精度。基于开源图像-文本编码器,实验表明该方法在密集与稀疏标题场景下分别实现约10%和7.7%的top-1检索提升,在top-k食材级检索中达到约28.6%的增益。此外,经文本融合增强的小模型(如ViT-B/32)可匹敌甚至超越大模型(如ViT-H, ViT-G, ViT-bigG),凸显其在资源受限环境下的有效性。代码与测试数据集将公开于:https://github.com/mailcorahul/f4-its

原文摘要 · Abstract (English)

The proliferation of digital food content has intensified the need for robust and accurate systems capable of fine-grained visual understanding and retrieval. In this work, we address the challenging task of food image-to-text matching, a critical component in applications such as dietary monitoring, smart kitchens, and restaurant automation. We propose F4-ITS: Fine-grained Feature Fusion for Food Image-Text Search, a training-free, vision-language model (VLM)-guided framework that significantly improves retrieval performance through enhanced multi-modal feature representations. Our approach introduces two key contributions: (1) a uni-directional(and bi-directional) multi-modal fusion strategy that combines image embeddings with VLM-generated textual descriptions to improve query expressiveness, and (2) a novel feature-based re-ranking mechanism for top-k retrieval, leveraging predicted food ingredients to refine results and boost precision. Leveraging open-source image-text encoders, we demonstrate substantial gains over standard baselines - achieving ~10% and ~7.7% improvements in top-1 retrieval under dense and sparse caption scenarios, and a ~28.6% gain in top-k ingredient-level retrieval. Additionally, we show that smaller models (e.g., ViT-B/32) can match or outperform larger counterparts (e.g., ViT-H, ViT-G, ViT-bigG) when augmented with textual fusion, highlighting the effectiveness of our method in resource-constrained settings. Code and test datasets will be made publicly available at: https://github.com/mailcorahul/f4-its

图文检索食物识别特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。