提出新框架提升遥感图像变化描述准确性
HIMEC: Directional Change Representation and Fixed-Interface Decoding for Remote Sensing Image Change Captioning

- 分三路提取变化特征,用查询编码器生成条件化记忆
- 固定接口解码使训练与推理一致,提升稳定性
- 在LEVIR-CC和SECOND-CC上显著优于基线方法
遥感图像变化描述(RSICC)将双时相影像转换为描述语义变化的句子。现有方法多直接以融合视觉特征作为解码器输入,忽略了中间变化结构与解码接口一致性。本文提出HIMEC,结合方向性变化表征(DCR)与固定接口解码。DCR在融合前将有符号差异分为外观导向、消失导向和共享上下文三路。学习型查询编码器将融合表示转为视觉条件化的改变查询令牌,作为场景解码器唯一的样本依赖记忆。训练阶段引入辅助短语解码器提供基于文本的监督。通过固定零输入,场景解码器在训练与推理中保持相同接口。此外,评估了基于教师强制局部状态的级联与推理时自回归状态的匹配性。在改变的LEVIR-CC验证对上,这些状态平均余弦距离为0.69。匹配模式的条件可恢复大部分性能损失,而状态对应随机打乱则无明显惩罚。该发现限于所评估级联。三种子对比下,HIMEC在LEVIR-CC上达到142.81±0.60的CIDEr得分,优于直接融合特征记忆的139.51±3.40。在SECOND-CC上,固定零与匹配条件分别达75.67与76.99 CIDEr,远超不匹配级联的60.77。源代码将在发表后公开于https://github.com/ayshaashra/HIMEC。
原文摘要 · Abstract (English)
Remote sensing image change captioning (RSICC) converts bitemporal imagery into a sentence describing semantic changes. Most RSICC methods condition caption decoders directly on fused visual features, leaving intermediate change structure and decoder-interface consistency less studied. We present HIMEC, combining Directional Change Representation (DCR) with fixed-interface decoding. DCR separates signed differences into appearance-oriented, disappearance-oriented, and shared-context streams before fusion. A learned-query encoder converts the fused representation into visually conditioned change-query tokens that form the scene decoder's only sample-dependent memory. A training-only auxiliary phrase decoder supplies caption-derived supervision. With a fixed zero input, the scene decoder maintains the same interface during training and inference. Separately, we evaluate a local-to-scene cascade conditioned on teacher-forced local states during training and autoregressive states at inference. On changed LEVIR-CC validation pairs, these states have a mean cosine distance of 0.69. Regime-matched conditioning recovers most of the associated deficit, whereas permuting state correspondence causes no detectable penalty. These findings are limited to the evaluated cascade. In a matched three-seed comparison, HIMEC reaches a Consensus-based Image Description Evaluation (CIDEr) score of $142.81\pm0.60$ on LEVIR-CC, versus $139.51\pm3.40$ for direct fused-feature memory. On SECOND-CC, fixed-zero and regime-matched diagnostic conditioning reach 75.67 and 76.99 CIDEr, respectively, versus 60.77 for the mismatched cascade. The source code will be made publicly available at https://github.com/ayshaashra/HIMEC upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。