用证据驱动框架帮科研软件工程师选对AI模型,提升可复现性。
Evidence-Driven Decision Support for AI Model Selection in Research Software Engineering
- 把模型选择转为多准则决策问题,结合知识图谱与自动化数据流。
- 50个真实案例验证,推荐结果与专家判断高度一致。
- 适合关注可解释性、可复现性的科研软件开发者使用。
人工智能模型的快速涌现给研究软件工程师在复杂研究工作流中选择、集成和维护合适模型带来了日益增长的挑战。当前模型选择常依赖碎片化元数据和个体经验,影响可复现性、透明度与整体软件质量。本文提出一种结构化、证据驱动的模型选择支持方法,将模型选择建模为多准则决策(MCDM)问题,构建集成自动化数据收集管道、结构化知识图谱与MCDM原则的决策支持框架(ModelSelect)。基于设计科学研究方法,该框架通过50个真实案例及与主流生成式AI系统的对比实验进行实证验证。结果表明,ModelSelect能生成可靠、可解释且可复现的推荐结果,与专家推理高度一致。在模型与库推荐任务中均实现高覆盖率和强理由对齐,性能媲美生成式AI助手,同时具备更优的可追溯性与一致性。本研究为科研软件工程中的透明、可复现决策支持奠定了严谨基础,提供了一条可扩展、可解释的路径,将实证证据融入AI模型推荐过程,从而提升研究软件决策的质量与稳健性。
原文摘要 · Abstract (English)
The rapid proliferation of artificial intelligence (AI) models and methods presents growing challenges for research software engineers and researchers who must select, integrate, and maintain appropriate models within complex research workflows. Model selection is often performed in an ad hoc manner, relying on fragmented metadata and individual expertise, which can undermine reproducibility, transparency, and overall research software quality. This work proposes a structured and evidence-driven approach to support AI model selection that aligns with both technical and contextual requirements. We conceptualize AI model selection as a Multi-Criteria Decision-Making (MCDM) problem and introduce an evidence-based decision-support framework that integrates automated data collection pipelines, a structured knowledge graph, and MCDM principles. Following the Design Science Research methodology, the proposed framework (ModelSelect) is empirically validated through 50 real-world case studies and comparative experiments against leading generative AI systems. The evaluation results show that ModelSelect produces reliable, interpretable, and reproducible recommendations that closely align with expert reasoning. Across the case studies, the framework achieved high coverage and strong rationale alignment in both model and library recommendation tasks, performing comparably to generative AI assistants while offering superior traceability and consistency. By framing AI model selection as an MCDM problem, this work establishes a rigorous foundation for transparent and reproducible decision support in research software engineering. The proposed framework provides a scalable and explainable pathway for integrating empirical evidence into AI model recommendation processes, ultimately improving the quality and robustness of research software decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。