用视觉语言模型实现文本与点击联合选择物体和材质,支持更灵活的图像编辑。
MAOAM: Unified Object and Material Selection with Vision-Language Models

- 基于视觉语言模型与分割头,统一处理物体和材质的文本与点击选择。
- 在真实与合成图像上生成带材质描述的数据,提升对材料的理解能力。
- 训练后可融合文本与点击输入,实现更自然的交互式编辑体验。
选择是交互式图像编辑的核心操作。为实现实用化,用户应能通过文本或点击指定并消除选择区域的歧义,且系统需支持对象及材料等多类标准的选择。材料级选择对表面重纹理或特定材料实例编辑具有重要意义。然而,现有基于视觉语言模型(VLM)的选择方法以对象为中心,通常仅支持单一交互模态,限制了应用范围。本文提出统一对象与材质选择框架MAOAM,可在文本和点击交互下实现像素级精确选择。MAOAM利用带有分割头的VLM,从用户提示中生成掩码:VLM解析用户意图(对象或材质级别),编码视觉实体、属性与空间关系;分割头将输出标记解码为掩码。关键挑战在于缺乏带文本标注的材料选择数据集。为此,我们设计可扩展的数据生成流程:收集含材料掩码的真实与合成图像,并借助VLM生成富含视觉语义的材料描述。我们在点击与文本选择的多任务目标上训练MAOAM,同时引入基于材料描述的辅助视觉问答任务,以促进深层材料理解。尽管训练时仅使用单模态提示,模型在推理时结合文本与点击可涌现性能提升,支持灵活的图像编辑工作流。实验表明,其在多样化的对象、材料与交互场景下均能实现准确连贯的选择,展现了实际应用中的鲁棒性。
原文摘要 · Abstract (English)
Selection is a core operation in interactive image editing. To be practical, a user should be able to specify and disambiguate the desired selection region through either text or click-based interactions, and the system should support selecting not only objects but also other criteria, such as materials. Material-based selection is valuable for tasks like re-texturing surfaces or editing instances of a specific material. However, existing vision-language-model (VLM) based selection methods are object-centric and typically support a single interaction modality, limiting their applicability. In this work, we thus present Mask Any Object And Material (MAOAM), a unified selection framework that enables precise object and material-level selection across both text- and click-based interactions. MAOAM leverages a VLM with a segmentation head to produce pixel-accurate masks from user prompts: the VLM interprets the user's selection intent (object or material-level) and encodes visual entities, attributes, and spatial relations, while the segmentation head decodes the output token into a mask. A key challenge is the lack of material selection datasets with text annotations. We propose a scalable data generation pipeline: we collect real and synthetic images with material masks, and leverage VLMs to generate material descriptions with rich visual-semantics. We train MAOAM with a multi-task objective over click and text-based selection, along with an auxiliary VQA task derived from the material descriptions to facilitate deeper material understanding. Despite being trained with uni-modal prompts, our model exhibits an emergent improvement in selection when combining text and clicks at inference, enabling flexible image editing workflows. Experiments demonstrate accurate and coherent selections across diverse objects, materials, and interaction scenarios, highlighting robustness in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。