用多头LoRA合成高保真少样本图像,提升细粒度分类性能。
ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets
- 将适配器分为类别共享与单图专属两部分,兼顾整体语义与细节特征。
- 在少样本数据下合成图像,下游分类准确率显著提升。
- 适合解决长尾分布、数据稀缺的细粒度识别问题。
除通用识别任务外,特定领域和细粒度场景常面临数据稀缺,尤其对尾部类别。为在数据稀缺下获得更少偏差、更可靠的模型,研究者利用扩散模型补充真实数据中代表性不足的区域。近期工作通过在少量真实图像上微调预训练扩散模型的LoRA来生成额外图像。然而,单图级LoRA虽能捕捉精细细节但多样性有限;类级LoRA虽具多样性(编码类别先验),却易忽略细节。为此,本文将适配器分离为类别共享的LoRA $A$(编码类别先验)与每张图像专属的LoRAs $/mathcal{B}$(编码图像特异性)。为增强共享LoRA $A$中的语义一致性,提出训练时保留类别边界框的语义增强策略。生成时,通过狄利克雷分布采样系数混合 $A$ 与 $/mathcal{B}$。在多个数据集上,合成图像兼具多样性与细节丰富性,且与真实少样本分布高度一致,显著提升下游分类精度。
原文摘要 · Abstract (English)
Beyond general recognition tasks, specialized domains and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to supplement underrepresented regions of real data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While an image-wise LoRA trained on a single image captures fine-grained details yet offers limited diversity, a class-wise LoRA trained over all shots produces diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into a class-shared LoRA~$A$ for class priors and per-image LoRAs~$\mathcal{B}$ for image-specific characteristics. To expose coherent class semantics in the shared LoRA~$A$, we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose $A$ with a mixture of $\mathcal{B}$ using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images are both diverse and detail-rich while closely aligning with the few-shot real distribution, yielding robust gains in downstream classification accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。