用字节偏移索引加速百亿级化学数据整合，效率提升740倍。
Accelerating Large-Scale Cheminformatics Using a Byte-Offset Indexing Architecture for Terabyte-Scale Data Integration
- 采用字节偏移索引架构，将复杂度从O(N×M)降至O(N+M)
- 17600万条数据中发现InChIKey哈希冲突，改用完整InChI
- 适用于需唯一性保障的超大规模科学数据集成场景
大规模化学数据库的整合是现代化学生物信息学研究的关键瓶颈，尤其对需要高质量多源验证数据的机器学习任务而言。本文以整合PubChem（1.76亿化合物）、ChEMBL和eMolecules三大公开库为例，构建用于分子性质预测的规范化数据集。研究探讨字节偏移索引是否能在百亿规模下突破暴力搜索的可扩展性极限，并保持数据完整性。结果表明，从预估需100天的暴力搜索算法，转变为字节偏移索引架构后，仅用3.2小时完成，性能提升740倍，算法复杂度由O(N×M)降至O(N+M)。对1.76亿条数据库条目系统性验证发现InChIKey存在哈希冲突，因此重构流程使用无冲突的完整InChI字符串。本文提供了性能基准，量化存储开销与科学严谨性的权衡，并对比了其他大规模集成策略。最终系统成功提取435,413个已验证化合物，展示了在唯一性约束超越哈希标识能力时的通用化科学数据集成方法。
原文摘要 · Abstract (English)
The integration of large-scale chemical databases represents a critical bottleneck in modern cheminformatics research, particularly for machine learning applications requiring high-quality, multi-source validated datasets. This paper presents a case study of integrating three major public chemical repositories: PubChem (176 million compounds), ChEMBL, and eMolecules, to construct a curated dataset for molecular property prediction. We investigate whether byte-offset indexing can practically overcome brute-force scalability limits while preserving data integrity at hundred-million scale. Our results document the progression from an intractable brute-force search algorithm with projected 100-day runtime to a byte-offset indexing architecture achieving 3.2-hour completion - a 740-fold performance improvement through algorithmic complexity reduction from $O(N \times M)$ to $O(N + M)$. Systematic validation of 176 million database entries revealed hash collisions in InChIKey molecular identifiers, necessitating pipeline reconstruction using collision-free full InChI strings. We present performance benchmarks, quantify trade-offs between storage overhead and scientific rigor, and compare our approach with alternative large-scale integration strategies. The resulting system successfully extracted 435,413 validated compounds and demonstrates generalizable principles for large-scale scientific data integration where uniqueness constraints exceed hash-based identifier capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。