用大模型智能区分前景背景,实现无数据集的精准物体移除
EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal
- 用多模态大模型识别目标、非目标前景和干净背景
- 通过分型聚合重建背景,使画面更真实且减少误生成
- 无需训练数据,比现有方法更少错误重绘,适合图像编辑场景
物体移除需防止被遮挡目标重现,并以结构与上下文一致性重建背景,而非仅填平空洞。现有无数据集方法通过修改扩散模型自注意力机制来避免引用被遮区域,但存在两大缺陷:(i) 将遮蔽区域视为唯一前景,误将非目标物体当作背景重新生成;(ii) 采用统一注意力约束,未区分多样背景类型,导致纹理模糊与结构错位。二者根源在于缺乏显式背景感知推理。本文提出EraseLoRA,一种无数据集框架,以背景感知推理替代注意力手术,并支持测试时适配。第一阶段(BFE)利用多模态大语言模型从单张图像-掩码对中分离目标前景、非目标前景与干净背景;第二阶段(BRSA)进行测试时优化,将推断出的背景子类型视为互补组件,通过重建与对齐目标强制其一致融合,无需显式干预注意力。作为可适配多种扩散模型的通用插件,EraseLoRA在背景重建上至少比先前无数据集方法提升23%真实度,同时几乎将不必要的前景重生成率减半,超越所有依赖训练数据的方法,且无需任何训练数据。代码已公开于https://shjo-april.github.io/EraseLoRA。
原文摘要 · Abstract (English)
Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches manipulate the diffusion model's internal self-attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a multimodal large-language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives without explicit attention intervention. As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。