测试大模型补全电商缺损图文信息的能力,发现效果受品类和模型规模影响大。
Benchmarking Multimodal Large Language Models for Missing Modality Completion in Product Catalogues
- 构建双子基准测试,评估图文互补质量与推荐效果。
- 大模型能抓语义但难对齐词级与像素级细节,性能不随模型变大而提升。
- 适合做电商内容修复、多模态生成研究者参考。
电商平台中缺失的模态信息(如缺图或缺描述)常由标注错误或元数据不全导致,影响商品展示及推荐系统等下游应用。本文基于近期多模态大模型(MLLMs)的生成能力,探究一个基础但未被充分研究的问题:MLLM能否在电商场景中生成缺失的模态?为此,我们提出了缺失模态产品补全基准(MMPCBench),包含内容质量补全与推荐效果两个子基准。我们在九个真实电商类别中评估了来自Qwen2.5-VL和Gemma-3系列的六款先进MLLM,在图像到文本与文本到图像补全任务上进行测试。结果表明,尽管模型能捕捉高层语义,但在词级与像素/块级对齐方面表现不佳。不同品类间性能差异显著,且模型规模与性能无明显正相关关系,与主流基准趋势相反。我们还探索了组相对策略优化(GRPO)以更好适配该任务,其提升了图像到文本补全表现,但未改善文本到图像补全。总体而言,这些发现揭示了当前MLLM在真实跨模态生成中的局限性,为更有效的缺失模态补全迈出了第一步。
原文摘要 · Abstract (English)
Missing-modality information on e-commerce platforms, such as absent product images or textual descriptions, often arises from annotation errors or incomplete metadata, impairing both product presentation and downstream applications such as recommendation systems. Motivated by the multimodal generative capabilities of recent Multimodal Large Language Models (MLLMs), this work investigates a fundamental yet underexplored question: can MLLMs generate missing modalities for products in e-commerce scenarios? We propose the Missing Modality Product Completion Benchmark (MMPCBench), which consists of two sub-benchmarks: a Content Quality Completion Benchmark and a Recommendation Benchmark. We further evaluate six state-of-the-art MLLMs from the Qwen2.5-VL and Gemma-3 model families across nine real-world e-commerce categories, focusing on image-to-text and text-to-image completion tasks. Experimental results show that while MLLMs can capture high-level semantics, they struggle with fine-grained word-level and pixel- or patch-level alignment. In addition, performance varies substantially across product categories and model scales, and we observe no trivial correlation between model size and performance, in contrast to trends commonly reported in mainstream benchmarks. We also explore Group Relative Policy Optimization (GRPO) to better align MLLMs with this task. GRPO improves image-to-text completion but does not yield gains for text-to-image completion. Overall, these findings expose the limitations of current MLLMs in real-world cross-modal generation and represent an early step toward more effective missing-modality product completion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。