无需训练即可清除各种遮挡，支持未知障碍物的实时修复。
Instruct2See: Learning to Remove Any Obstructions Across Distributions
- 用多模态提示统一建模遮挡，通过交叉注意力增强理解。
- 在分布内和分布外障碍上均表现稳定，零样本泛化能力强。
- 适合需要快速处理复杂遮挡的视觉系统，如自动驾驶、监控场景。
图像常因拍摄限制被各类障碍物遮挡，影响对目标物体的观测。现有方法多针对特定遮挡（如栅栏、雨滴），但真实世界遮挡类型繁多，全面采集数据不现实。为此，我们提出Instruct2See，一种新型零样本框架，可处理已见与未见的遮挡。核心思路是将遮挡去除统一为软-硬掩码恢复问题，利用视觉语义和文本指令等多模态提示，经交叉注意力单元提升上下文理解与模式控制能力。此外，可调掩码适配器实现动态软掩码，支持对不准确掩码的实时修正。大量实验表明，Instruct2See在分布内和分布外遮挡上均表现出强性能与良好泛化能力，无论遮挡是否出现在训练阶段。代码与数据集见 https://jhscut.github.io/Instruct2See。
原文摘要 · Abstract (English)
Images are often obstructed by various obstacles due to capture limitations, hindering the observation of objects of interest. Most existing methods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, making comprehensive data collection impractical. To overcome these challenges, we propose Instruct2See, a novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating it as a soft-hard mask restoration problem, where any obstruction can be represented using multi-modal prompts, such as visual semantics and textual instructions, processed through a cross-attention unit to enhance contextual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjustment of inaccurate masks. Extensive experiments on both in-distribution and out-of-distribution obstacles show that Instruct2See consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during the training phase. Code and dataset are available at https://jhscut.github.io/Instruct2See.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。