让AI学会预测物体在场景中合理出现的位置,提升虚拟生成与编辑效果。
Imagining the Unseen: Generative Location Modeling for Object Placement
- 用自回归Transformer建模图像和物体类别的位置分布,处理多解性难题。
- 在OPA数据集上定位准确率优于判别式模型和传统拼接方法。
- 适用于虚拟场景生成、物体插入等下游任务,尤其适合内容创作人员。
位置建模,即判断不存在的物体在场景中可能合理出现的位置,可广泛应用于自动物体插入、虚拟现实场景构建等计算机视觉任务,但该方向仍鲜有研究。本文提出一种生成式位置模型,给定物体类别和图像后,学习预测其合理边界框。模型先对图像和目标物体类别进行分词,再通过自回归Transformer解码坐标。该方法有效应对两大挑战:合理位置的固有“一对多”特性,以及现有数据集标注稀疏(有效位置标注不足1%)。此外,引入直接偏好优化(DPO)利用负样本标签,优化空间预测。实验证明,本模型在OPA数据集上的定位精度优于判别式基线与图像合成方法。进一步测试显示,在物体插入任务中,模型为现成的修复模型提供位置建议,生成结果在视觉连贯性上优于当前最先进的指令微调编辑方法,证实了高精度位置模型在下游应用中的价值。
原文摘要 · Abstract (English)
Location modeling, or determining where non-existing objects could feasibly appear in a scene, has the potential to benefit numerous computer vision tasks, from automatic object insertion to scene creation in virtual reality. Yet, this capability remains largely unexplored to date. In this paper, we develop a generative location model that, given an object class and an image, learns to predict plausible bounding boxes for such an object. Our approach first tokenizes the image and target object class, then decodes bounding box coordinates through an autoregressive transformer. This formulation effectively addresses two core challenges in locatio modeling: the inherent one-to-many nature of plausible locations, and the sparsity of existing location modeling datasets, where fewer than 1% of valid placements are labeled. Furthermore, we incorporate Direct Preference Optimization to leverage negative labels, refining the spatial predictions. Empirical evaluations reveal that our generative location model achieves superior placement accuracy on the OPA dataset as compared to discriminative baselines and image composition approaches. We further test our model in the context of object insertion, where it proposes locations for an off-the-shelf inpainting model to render objects. In this respect, our proposal exhibits improved visual coherence relative to state-of-the-art instruction-tuned editing methods, demonstrating a high-performing location model's utility in a downstream application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。