LIT-PCBA基准存在严重数据泄露,导致模型虚假表现。
Data Leakage and Redundancy in the LIT-PCBA Benchmark
- 发现训练集与验证集间存在大量分子重复和相似结构
- 2,491个完全相同的非活性分子同时出现在训练和验证集
- 零样本模型也受泄漏影响,无法真实评估泛化能力
LIT-PCBA被广泛用于虚拟筛选模型的基准测试,但我们的审计发现其存在根本性缺陷。在各划分中存在广泛的数据泄露与分子冗余,包括同一分区及跨分区的二维相同配体、普遍存在的类似物重叠以及低多样性查询集。例如,在ALDH1中,有323个活性化合物的训练-验证对在ECFP4 Tanimoto相似度≥0.6下重复;在所有靶点中,2,491个二维相同的非活性分子同时出现在训练和验证集中,且对应活性分子极少。这些重叠使模型可通过骨架记忆而非泛化能力取得高分,从而虚增富集因子与AUROC值。这些问题并非偶然——一个无参数的纯记忆基线即可利用这些漏洞达到甚至超过现有深度学习与3D相似性模型的报告性能。因此,几乎所有在LIT-PCBA上发表的结果均不可靠。即使在‘零样本’模式下评估的模型也受类似物泄漏影响,削弱了泛化能力声称。当前形式的基准无法衡量模型发现新化学类型的能力,不应被视为方法进步的证据。所有代码、数据与基线实现均开源于:https://github.com/sievestack/LIT-PCBA-audit
原文摘要 · Abstract (English)
LIT-PCBA is widely used to benchmark virtual screening models, but our audit reveals that it is fundamentally compromised. We find extensive data leakage and molecular redundancy across its splits, including 2D-identical ligands within and across partitions, pervasive analog overlap, and low-diversity query sets. In ALDH1 alone, for instance, 323 active training -- validation analog pairs occur at ECFP4 Tanimoto similarity $\geq 0.6$; across all targets, 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. These overlaps allow models to succeed through scaffold memorization rather than generalization, inflating enrichment factors and AUROC scores. These flaws are not incidental -- they are so severe that a trivial memorization-based baseline with no learnable parameters can exploit them to match or exceed the reported performance of state-of-the-art deep learning and 3D-similarity models. As a result, nearly all published results on LIT-PCBA are undermined. Even models evaluated in "zero-shot" mode are affected by analog leakage into the query set, weakening claims of generalization. In its current form, the benchmark does not measure a model's ability to recover novel chemotypes and should not be taken as evidence of methodological progress. All code, data, and baseline implementations are available at: https://github.com/sievestack/LIT-PCBA-audit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。