用大模型辅助系统性映射研究,提升效率但需警惕幻觉和人工校验。
On the Use of a Large Language Model to Support the Conduction of a Systematic Mapping Study: A Brief Report from a Practitioner's View
- 借助大模型自动完成文献筛选与数据提取,减少重复劳动。
- 显著缩短任务耗时,提升数据提取标准化程度。
- 适合希望提速但需谨慎应对幻觉与提示工程挑战的研究者。
大语言模型(LLMs)在科学界引发广泛关注,因其能处理海量文本并支持证据综合。尽管已有研究指出其可加速系统性综述中的筛选与数据提取步骤,但关于其在全流程中实际应用的详细报告仍较少。本文基于实践,分享了在系统性映射研究中使用大模型的经验,涵盖实施步骤、必要调整及主要挑战。正面效果包括:(i) 显著减少重复性任务的时间消耗;(ii) 提升数据提取的一致性。负面问题包括:(i) 构建可靠且结构清晰的提示需大量迭代,尤其对新手而言可能抵消预期省时优势;(ii) 存在幻觉风险;(iii) 必须持续进行人工验证。本文提出实用建议,帮助研究人员评估大模型在系统性映射与综述中的效率增益与方法论风险。
原文摘要 · Abstract (English)
The use of Large Language Models (LLMs) has drawn growing interest within the scientific community. LLMs can handle large volumes of textual data and support methods for evidence synthesis. Although recent studies highlight the potential of LLMs to accelerate screening and data extraction steps in systematic reviews, detailed reports of their practical application throughout the entire process remain scarce. This paper presents an experience report on the conduction of a systematic mapping study with the support of LLMs, describing the steps followed, the necessary adjustments, and the main challenges faced. Positive aspects are discussed, such as (i) the significant reduction of time in repetitive tasks and (ii) greater standardization in data extraction, as well as negative aspects, including (i) considerable effort to build reliable well-structured prompts, especially for less experienced users, since achieving effective prompts may require several iterations and testing, which can partially offset the expected time savings, (ii) the occurrence of hallucinations, and (iii) the need for constant manual verification. As a contribution, this work offers lessons learned and practical recommendations for researchers interested in adopting LLMs in systematic mappings and reviews, highlighting both efficiency gains and methodological risks and limitations to be considered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。