让场景图自动补关系、修冲突，生成更合理的视觉内容。
SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation
- 用自回归模型预测物体间无冲突的关系，支持增删节点与修改边。
- 在多个数据集上显著优于基线方法，尤其在复杂关系推理上表现突出。
- 适合需要精准语义理解的图像生成和机器人交互任务使用。
场景图捕捉物体间的复杂关系，是内容生成与操作的强大先验。然而，合理地操纵场景图——无论是添加节点还是修改边——仍是一个极具挑战且未被充分探索的任务。诸如添加新节点或推理其与其他节点关系等操作，在计算上难以处理，因为单一边的修改就可能因图中复杂的依赖关系引发冲突。为此，我们提出SG-Tailor，一种自回归模型，用于预测任意两个节点之间的无冲突关系。SG-Tailor不仅能推断物体间的关系，包括为新增节点生成常识性边，还能通过解决边修改带来的冲突，生成一致连贯的操控后图谱，适用于下游任务。对于节点添加，模型查询目标节点与其他节点以预测合适关系；对于边修改，采用“剪切-拼接”策略解决冲突并全局调整图结构。大量实验表明，SG-Tailor显著优于现有方法，并可无缝作为插件模块集成至图像生成与机器人操作任务中。
原文摘要 · Abstract (English)
Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。