数据筛选在小样本环境下反而加剧模型坍缩,需用协作代理参考缓解。
When Sample Selection Bias Precipitates Model Collapse

- 用局部参考筛选数据时,因信息不全导致选择偏差。
- 碎片化数据下筛选加速模型坍缩,多样性按幂律衰减。
- 无需共享原始数据,通过水桶距离构建跨域代理参考。
合成数据的递归训练可缓解数据稀缺,但可能导致模型坍缩——重复训练会削弱分布尾部特征并使输出趋同。数据筛选常被视为解决方法,但其可靠性取决于验证器所用参考分布。在低资源验证场景中,每个验证器仅能观测目标流形的一小部分、零散且有偏的数据,此时筛选本身即产生偏差。该情况常见于医疗联盟或私有金融机构等数据孤岛,原始数据无法集中,本地参考天然不完整。结果,筛选优先保留与本地流形一致的样本,而剔除全局关键的尾部模式,使筛选从防范坍缩的机制转变为促发坍缩的诱因。我们理论证明此类孤岛式筛选会加速坍缩,并引发幂律多样性衰减。作为初步缓解方案,我们构建了无需共享原始数据的水桶距离代理参考。实验表明,本地参考筛选在偏斜分布上失效,而协作代理参考能有效缓解多样性退化,提示在真实数据覆盖稀疏或碎片化的场景中,递归合成数据流程需格外谨慎。
原文摘要 · Abstract (English)
The proliferation of recursive training on synthetic data can alleviate data scarcity but risks model collapse, where repeated training erodes distributional tails and homogenizes outputs. Data selection is widely viewed as a remedy, yet its reliability depends critically on the reference distribution used by the verifier. We show that in low-resource verification regimes, where each verifier observes only a small, fragmented, and biased slice of the target manifold, selection itself becomes biased. This situation naturally arises in low-resource data silos such as healthcare consortia or proprietary financial institutions, where raw data cannot be pooled and local references are inherently incomplete. As a result, selection preferentially retains samples aligned with the local manifold while pruning globally relevant tail modes, turning from a safeguard against collapse into a mechanism that precipitates it. We theoretically prove that such siloed selection accelerates collapse and induces power-law diversity decay. As an initial mitigation, we construct Wasserstein proxy references from multiple silos without sharing raw data. Empirical results confirm that local-reference selection fails on skewed distributions, whereas collaborative proxy references mitigate diversity degradation, suggesting that recursive synthetic-data pipelines require particular caution when real-data coverage is fragmented or scarce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。