将双时相图像视为视频,用轻量级模型统一解决变化检测与描述任务。
Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective
- 把两幅时间图像看作视频帧,加入可学习感知帧增强差异捕捉
- 在8个基准上超越主流方法,参数仅13%、计算量34%即达顶尖性能
- 无需复杂模块,适合想简化架构的研究者或工程应用
本文提出Change3D框架,从视频建模视角重新定义变化检测与描述任务。现有方法将双时相图像视为独立图像,使用共享权重编码器提取特征后通过专用变化提取器捕捉差异,但特征编码缺乏对变化区域的关注,且不同任务需不同提取器,难统一。Change3D将双时相图像视为两个视频帧,插入可学习感知帧,使视频编码器直接交互并感知差异,从而摆脱复杂变化提取器,实现多任务统一框架。我们在多个任务(包括二值变化检测、语义变化检测和建筑损毁评估)和8个标准基准上验证该方法。不依赖复杂组件,仅用约6%-13%参数量、8%-34%浮点运算量的超轻量视频模型,即可达到领先性能。我们希望Change3D能成为2D模型的替代方案,推动后续研究。
原文摘要 · Abstract (English)
In this paper, we present Change3D, a framework that reconceptualizes the change detection and captioning tasks through video modeling. Recent methods have achieved remarkable success by regarding each pair of bi-temporal images as separate frames. They employ a shared-weight image encoder to extract spatial features and then use a change extractor to capture differences between the two images. However, image feature encoding, being a task-agnostic process, cannot attend to changed regions effectively. Furthermore, different change extractors designed for various change detection and captioning tasks make it difficult to have a unified framework. To tackle these challenges, Change3D regards the bi-temporal images as comprising two frames akin to a tiny video. By integrating learnable perception frames between the bi-temporal images, a video encoder enables the perception frames to interact with the images directly and perceive their differences. Therefore, we can get rid of the intricate change extractors, providing a unified framework for different change detection and captioning tasks. We verify Change3D on multiple tasks, encompassing change detection (including binary change detection, semantic change detection, and building damage assessment) and change captioning, across eight standard benchmarks. Without bells and whistles, this simple yet effective framework can achieve superior performance with an ultra-light video model comprising only ~6%-13% of the parameters and ~8%-34% of the FLOPs compared to state-of-the-art methods. We hope that Change3D could be an alternative to 2D-based models and facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。