用语义分割任务评估可见光与红外图像融合效果
Rethinking the Evaluation of Visible and Infrared Image Fusion
- 以分割任务为基准，利用标签数据评估融合图像质量
- 发现多数融合方法性能不如直接使用可见光图像
- 提出可替代的评估指标，适合无标签场景使用
可见光与红外图像融合（VIF）在目标检测、语义分割等高级视觉任务中受到广泛关注。然而，由于缺乏真实标签，VIF方法的评估仍具挑战性。本文提出一种面向分割的评估方法（SEA），通过引入最新的VIF数据集中的分割标签，利用通用分割模型对融合图像进行预测并对比真实标签来评估性能。实验表明，尽管近半数红外图像表现优于可见光图像，但多数融合方法的性能仍不及直接使用可见光图像。进一步分析发现，传统评价指标中的梯度融合度 $Q_{ ext{ABF}}$ 和视觉信息保真度 $Q_{ ext{VIFF}}$ 与SEA结果相关性最高，可在无分割标签时作为有效代理指标。代码已开源。
原文摘要 · Abstract (English)
Visible and Infrared Image Fusion (VIF) has garnered significant interest across a wide range of high-level vision tasks, such as object detection and semantic segmentation. However, the evaluation of VIF methods remains challenging due to the absence of ground truth. This paper proposes a Segmentation-oriented Evaluation Approach (SEA) to assess VIF methods by incorporating the semantic segmentation task and leveraging segmentation labels available in latest VIF datasets. Specifically, SEA utilizes universal segmentation models, capable of handling diverse images and classes, to predict segmentation outputs from fused images and compare these outputs with segmentation labels. Our evaluation of recent VIF methods using SEA reveals that their performance is comparable or even inferior to using visible images only, despite nearly half of the infrared images demonstrating better performance than visible images. Further analysis indicates that the two metrics most correlated to our SEA are the gradient-based fusion metric $Q_{\text{ABF}}$ and the visual information fidelity metric $Q_{\text{VIFF}}$ in conventional VIF evaluation metrics, which can serve as proxies when segmentation labels are unavailable. We hope that our evaluation will guide the development of novel and practical VIF methods. The code has been released in \url{https://github.com/Yixuan-2002/SEA/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。