用双智能体系统自动扩增实例分割数据,提升小样本类别性能
InstaDA: Augmenting Instance Segmentation Data with Dual-Agent System
- 引入文本与图像双智能体,协同优化提示词与生成新实例
- 在LVIS数据集上使框和掩码平均精度分别提升+4.0和+3.3
- 特别改善常见与频繁类别的识别,适合数据稀缺场景
高质量实例分割数据的获取因标注耗时且类别分布不均而困难。现有方法虽结合复制粘贴与扩散模型提升多样性,但大语言模型(LLMs)与扩散模型间协作不足,且未充分利用训练数据信息。为此,我们提出InstaDA——一种无需训练的双智能体系统,用于增强实例分割数据集。首先,设计文本智能体(T-Agent),通过LLM与扩散模型协同,采用创新的提示重思机制,基于生成图像迭代优化提示词,提升图像利用率并改进提示质量。其次,提出图像智能体(I-Agent),基于训练图像生成新实例,丰富数据分布。两个智能体独立自动化运行,保障实用性与效率。在LVIS 1.0验证集上的实验表明,InstaDA相比基线模型,框平均精度(AP)提升+4.0,掩码平均精度(AP)提升+3.3;优于领先模型DiverGen,框AP高+0.3,掩码AP高+0.1,其中常见类别框AP提升+0.7,常见类别掩码AP提升+0.2,频繁类别掩码AP提升+0.5。
原文摘要 · Abstract (English)
Acquiring high-quality instance segmentation data is challenging due to the labor-intensive nature of the annotation process and significant class imbalances within datasets. Recent studies have utilized the integration of Copy-Paste and diffusion models to create more diverse datasets. However, these studies often lack deep collaboration between large language models (LLMs) and diffusion models, and underutilize the rich information within the existing training data. To address these limitations, we propose InstaDA, a novel, training-free Dual-Agent system designed to augment instance segmentation datasets. First, we introduce a Text-Agent (T-Agent) that enhances data diversity through collaboration between LLMs and diffusion models. This agent features a novel Prompt Rethink mechanism, which iteratively refines prompts based on the generated images. This process not only fosters collaboration but also increases image utilization and optimizes the prompts themselves. Additionally, we present an Image-Agent (I-Agent) aimed at enriching the overall data distribution. This agent augments the training set by generating new instances conditioned on the training images. To ensure practicality and efficiency, both agents operate as independent and automated workflows, enhancing usability. Experiments conducted on the LVIS 1.0 validation set indicate that InstaDA achieves significant improvements, with an increase of +4.0 in box average precision (AP) and +3.3 in mask AP compared to the baseline. Furthermore, it outperforms the leading model, DiverGen, by +0.3 in box AP and +0.1 in mask AP, with a notable +0.7 gain in box AP on common categories and mask AP gains of +0.2 on common categories and +0.5 on frequent categories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。