通过精炼样本内信息,提升图文数据质量与对齐度。
Squeeze Out Tokens from Sample for Finer-Grained Data Governance
- 双分支架构分别优化图像和文本,提取关键区域与对象类别
- 在多个任务上优于现有方法,显著提升图文检索与视觉推理性能
- 适合需要高质量图文数据集的多模态模型训练场景
数据缩放定律表明,错误率随训练规模呈幂律下降,说明无差别扩展数据效益递减。因此,数据治理提出通过剔除低价值样本压缩数据集。然而,评估单个样本对模型性能的影响需遍历所有组合,计算成本过高。当前方法多依赖启发式评分筛选样本,但保留的样本仍含大量冗余内容。本文将数据治理从‘筛除’升级为‘榨取’,提出双分支数据榨汁器(DataJuicer),实现更细粒度的样本内治理:视觉分支保留显著图像块并提取相关对象类别,文本分支结合这些类别增强描述。实验表明,该方法在多个数据集上显著优于现有数据筛除法(DataSieve),在图文检索、分类及密集视觉推理任务中表现更优。
原文摘要 · Abstract (English)
Widely observed data scaling laws, in which error falls off as a power of the training size, demonstrate the diminishing returns of unselective data expansion. Hence, data governance is proposed to downsize datasets through pruning non-informative samples. Yet, isolating the impact of a specific sample on overall model performance is challenging, due to the vast computation required for tryout all sample combinations. Current data governors circumvent this complexity by estimating sample contributions through heuristic-derived scalar scores, thereby discarding low-value ones. Despite thorough sample sieving, retained samples contain substantial undesired tokens intrinsically, underscoring the potential for further compression and purification. In this work, we upgrade data governance from a 'sieving' approach to a 'juicing' one. Instead of scanning for least-flawed samples, our dual-branch DataJuicer applies finer-grained intra-sample governance. It squeezes out informative tokens and boosts image-text alignments. Specifically, the vision branch retains salient image patches and extracts relevant object classes, while the text branch incorporates these classes to enhance captions. Consequently, DataJuicer yields more refined datasets through finer-grained governance. Extensive experiments across datasets demonstrate that DataJuicer significantly outperforms existing DataSieve in image-text retrieval, classification, and dense visual reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。