提出自动评估商品图背景修复质量的新框架，减少人工标注成本。
An Evaluation Framework for Product Images Background Inpainting based on Human Feedback and Product Consistency
- 基于4.4万张图像的人类反馈训练奖励模型，判断背景合理性。
- 通过微调分割模型比对原始与生成图像的产品一致性，识别失真。
- 在开源模型中精度达96.4%，适合商品图像生成质量评估场景。
在商品广告应用中，利用AI技术自动修复商品图背景已成为重要任务。然而，现有方法常出现背景不恰当或商品不一致的问题，且传统评价方式与人类反馈不一致，导致评估依赖人工标注。为此，本文提出基于人类反馈与产品一致性的评估框架HFPC，包含两个模块：首先，基于BLIP多模态特征和对比学习，利用44,000张自动化修复商品图的人类反馈数据训练奖励模型，以判断背景合理性；其次，采用微调的分割模型分别提取原始与生成图像中的商品区域，并比较其差异以筛选出产品不一致的图像。大量实验表明，HFPC能有效评估生成图像质量，显著降低人工标注成本，且在与其他开源视觉质量评估模型对比中达到96.4%的最高精确率。相关数据集与代码已公开于https://github.com/created-Bi/background_inpainting_products_dataset。
原文摘要 · Abstract (English)
In product advertising applications, the automated inpainting of backgrounds utilizing AI techniques in product images has emerged as a significant task. However, the techniques still suffer from issues such as inappropriate background and inconsistent product in generated product images, and existing approaches for evaluating the quality of generated product images are mostly inconsistent with human feedback causing the evaluation for this task to depend on manual annotation. To relieve the issues above, this paper proposes Human Feedback and Product Consistency (HFPC), which can automatically assess the generated product images based on two modules. Firstly, to solve inappropriate backgrounds, human feedback on 44,000 automated inpainting product images is collected to train a reward model based on multi-modal features extracted from BLIP and comparative learning. Secondly, to filter generated product images containing inconsistent products, a fine-tuned segmentation model is employed to segment the product of the original and generated product images and then compare the differences between the above two. Extensive experiments have demonstrated that HFPC can effectively evaluate the quality of generated product images and significantly reduce the expense of manual annotation. Moreover, HFPC achieves state-of-the-art(96.4% in precision) in comparison to other open-source visual-quality-assessment models. Dataset and code are available at: https://github.com/created-Bi/background_inpainting_products_dataset
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。