用GPT-4零样本推断图像中物体的遮挡顺序，提升视觉理解鲁棒性。
GPT-4 for Occlusion Order Recovery
- 通过设计提示词引导GPT-4分析图像，推理物体遮挡顺序。
- 在COCOA和InstaOrder数据集上优于基线方法，无需标注数据训练。
- 利用语义、视觉模式与常识知识，可直接嵌入现有视觉系统。
遮挡仍是当前视觉模型在复杂密集真实场景中稳健理解的关键挑战。为解决此问题并准确预测物体间的遮挡顺序关系，我们提出利用预训练GPT-4模型的强大能力进行推断。通过向GPT-4输入图像并搭配特定设计的提示词，模型可分析图像并生成遮挡顺序预测结果。该输出可解析为遮挡矩阵，用于辅助其他遮挡处理任务与图像理解。我们在COCOA和InstaOrder数据集上评估了模型表现。结果显示，借助语义上下文、视觉模式与常识知识，模型能生成更精确的顺序预测。与基线方法不同，该模型可实现零样本推理，无需标注训练数据，且易于集成到现有遮挡处理框架中。
原文摘要 · Abstract (English)
Occlusion remains a significant challenge for current vision models to robustly interpret complex and dense real-world images and scenes. To address this limitation and to enable accurate prediction of the occlusion order relationship between objects, we propose leveraging the advanced capability of a pre-trained GPT-4 model to deduce the order. By providing a specifically designed prompt along with the input image, GPT-4 can analyze the image and generate order predictions. The response can then be parsed to construct an occlusion matrix which can be utilized in assisting with other occlusion handling tasks and image understanding. We report the results of evaluating the model on COCOA and InstaOrder datasets. The results show that by using semantic context, visual patterns, and commonsense knowledge, the model can produce more accurate order predictions. Unlike baseline methods, the model can reason about occlusion relationships in a zero-shot fashion, which requires no annotated training data and can easily be integrated into occlusion handling frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。