用可定制框架评估大模型在科研中的细粒度能力,无需人工标注。
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

- 基于知识图谱构建可调控的科学知识单元,实现数据自动映射。
- 通过多模型共识识别相关知识单元,支持高效精准的基准检索。
- 在化学与医疗领域验证有效,适合科研场景下的模型评估需求。
大语言模型(LLMs)在科学研究中应用日益广泛,但现有评估方法难以反映实际所需的细粒度能力。多数基准为人工构建或领域通用,限制了可扩展性与真实应用场景的契合度。本文提出新框架 SciCustom,支持从大规模科学数据中自定义构建评测基准,以评估特定应用下的科学能力。SciCustom 首先将科学知识组织为具有可控粒度的本体引导知识单元,并训练标签器将大规模数据实例映射至该知识空间。针对特定需求,通过基于投票的多模型共识识别相关知识单元。这些单元支持基于二分查找的相关性感知基准检索,随后进行代理子集选择与数据驱动的基准生成,实现高效评估。在化学与医疗领域的实验表明,SciCustom 能揭示标准基准忽略的模型细粒度差异,且无需专家标注或合成问题生成。该工作为大模型科学能力评估提供了可扩展、面向应用的基础。源代码已开源:https://github.com/yjwtheonly/SciCustom。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。