arXiv:2603.22018cs.LGcs.SE2026-03

构建首个生物信息学论文与代码一致性基准,助力科研可复现

Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics

  • 通过细粒度对齐论文句子与代码函数,构建高质量配对数据集
  • 提出统一跨模态框架,在三类任务中均实现强性能表现
  • 适合关注科研可复现性与软件可靠性的生物信息学研究者

确保论文与对应代码实现的一致性是保障科学发现可复现性和软件系统可靠性的重要前提。然而,这一问题在生物信息学领域长期未受重视,方法描述与实际代码之间普遍存在不一致。为此,本文提出论文-代码一致性检测新任务,旨在刻画论文方法描述与代码实现之间的跨模态语义对齐。在数据层面,构建首个生物信息学领域的基准数据集BioCon,包含48个生物信息学软件项目及其关联论文;该数据集通过句级方法描述与函数级代码片段的细粒度对齐,结合专家标注与难负样本采样策略,形成高质量句子-代码配对数据。在方法层面,提出统一的跨模态一致性检测框架,利用预训练模型联合编码论文句子与代码函数。从句级分类、跨模态检索和项目级一致性评估三个角度进行系统分析。实验表明,所提方法在一致性判别与语义对齐方面均表现优异。本工作首次建立系统化的论文-代码一致性分析基准与框架,开辟新研究方向,为提升生物信息学软件的可复现性与可靠性奠定基础。

原文摘要 · Abstract (English)

Ensuring consistency between research papers and their corresponding software code implementations is a fundamental prerequisite for guaranteeing the reproducibility of scientific findings and the reliability of software systems. However, this issue has received limited attention to date, particularly in the field of bioinformatics, where inconsistencies between methodological descriptions in papers and their actual code implementations are prevalent. To address this gap, we introduce a novel research task, namely paper-code consistency detection, which aims to characterize the cross-modal semantic alignment between methodological descriptions in papers and their corresponding code implementations. At the data level, we construct the first benchmark dataset for this task in the bioinformatics domain, termed BioCon, comprising 48 bioinformatics software projects and their associated publications. BioCon is built by fine-grained alignment between sentence-level methodological descriptions in papers and function-level code snippets, combined with expert annotation and hard negative sampling strategies, resulting in a high-quality sentence-code paired dataset. At the methodological level, we propose a unified cross-modal consistency detection framework that leverages pre-trained models to jointly encode paper sentences and code functions. We conduct a systematic analysis from three perspectives: sentence-level classification, cross-modal retrieval, and project-level consistency assessment. Experimental results demonstrate that the proposed approach achieves strong performance in both consistency discrimination and semantic alignment. Overall, this work establishes the first systematic benchmark and framework for paper-code consistency analysis, opening a new research direction and providing a foundation for improving reproducibility and reliability in bioinformatics software.

可复现性生物信息学跨模态基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。