arXiv:2512.01675cs.CV2025-12被引 2

解决图像生成中罕见类别的质量下降问题,提升合成数据的多样性与真实性。

GRASP: Guided Residual Adapters with Sample-wise Partitioning

  • 按条件样本静态分组,为每组设计专用残差适配器,增强梯度对齐。
  • 在医疗和ImageNet长尾数据集上,显著降低FID达80%,尾部类别覆盖提升44%。
  • 无需修改生成流程,可与采样策略组合使用,适合医学影像等长尾场景。

文本到图像的流匹配变压器在长尾设置下性能急剧下降:尾部类别的输出在保真度和多样性上均出现坍塌,限制了其作为稀有病症合成增广的价值。我们发现这是由于微调过程中头部与尾部梯度对齐不足所致,而现有的条件和采样干预无法解决此优化层面的问题。为此提出GRASP(带样本划分的引导残差适配器):一种确定性的条件空间划分,配合变压器前馈层中的组特定残差适配器,保持流匹配目标和采样器不变。在条件流匹配中,条件值对应不同的概率路径集合,沿条件进行划分是结构上正确的因子化方式,可作为梯度对齐的代理。由于划分是静态的,每个尾部样本都能确保更新其分配的专家,从而绕过极端长尾失效模式。关键的是,GRASP具有非侵入性和可组合性:在MIMIC-CXR-LT上,结合推理时自引导少数采样,获得最优全标签IRS表现,优于单一干预。仅用GRASP即能将整体FID降低80%,尾部覆盖率提升44%,优于全微调、学习路由MoE和少数类引导。将其用于下游DenseNet分类器在NIH-CXR-LT上的训练,显著优于所有非GRASP替代方案,宏平均F1达到真实数据水平,且在13个类别中有9个实现非零F1,而全微调仅3个。ImageNet-LT上的结果证实该机制不依赖医学先验知识。

原文摘要 · Abstract (English)

Text-to-image flow matching transformers degrade sharply in long-tail settings: tail-class outputs collapse in fidelity and diversity, limiting their value as synthetic augmentation for rare conditions. We trace this to low head-versus-tail gradient alignment during fine-tuning, an optimization-level pathology that conditioning- and sampling-side interventions do not address. We propose GRASP (Guided Residual Adapters with Sample-wise Partitioning): a deterministic partition of the conditioning space, paired with group-specific residual adapters in the transformer feedforward layers, that leaves the flow-matching objective and the sampler untouched. In conditional flow matching, condition values index distinct sets of probability paths, so partitioning along the conditioning is the structurally correct factorization suitable as gradient alignment proxy. Because the partition is static, every tail sample is guaranteed to update its assigned expert, which bypasses extreme longtail failure modes. Crucially, GRASP is non-invasive and composable: on MIMIC-CXR-LT, combining GRASP with self-guided minority sampling at inference time yields the best all-labels IRS we observe, beyond either intervention alone. GRASP itself reduces overall FID by up to 80\% and lifts tail-class coverage by up to 44\% over full fine-tuning, learned-routing MoE, and minority guidance. Used as training data for a downstream DenseNet classifier on NIH-CXR-LT, GRASP synthetics significantly outperform every non-GRASP alternative on macro F1, match the macro F1 obtained from real training data, and yield nonzero F1 on $9$ of $13$ classes versus $3$ of $13$ from full fine-tuning. Results on ImageNet-LT confirm the mechanism is not tied to medical inductive bias.

图像生成长尾分布扩散模型合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。