用大模型自动做科学文献综述，准确率超87%。
Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis
- 用提示工程和新损失函数微调大模型，结合检索增强生成。
- 微调后模型生成的摘要相关性达87.6%，无关内容从4.56%降至1.9%。
- 适合需要快速整合大量文献的研究者，尤其资源有限的团队。
本研究探索利用大语言模型（LLM）自动化科学文献的元分析过程。元分析是一种通过整合多篇研究论文来获得全面理解的统计方法，但传统人工操作耗时费力且易出错。为此，我们提出一种新方法：在大规模科学数据集上微调LLM，以应对大数据处理与结构化信息提取挑战。通过引入检索增强生成（RAG），并结合提示工程与新设计的损失函数——逆余弦距离（ICD），实现元分析流程的自动化与优化。人类评估结果显示，微调后的模型生成的元分析摘要相关性达到87.6%，上下文无关内容从4.56%下降至1.9%。实验在低资源环境下进行，证明该方法能显著提升元分析自动化效率与可靠性。
原文摘要 · Abstract (English)
This study investigates the automation of meta-analysis in scientific documents using large language models (LLMs). Meta-analysis is a robust statistical method that synthesizes the findings of multiple studies support articles to provide a comprehensive understanding. We know that a meta-article provides a structured analysis of several articles. However, conducting meta-analysis by hand is labor-intensive, time-consuming, and susceptible to human error, highlighting the need for automated pipelines to streamline the process. Our research introduces a novel approach that fine-tunes the LLM on extensive scientific datasets to address challenges in big data handling and structured data extraction. We automate and optimize the meta-analysis process by integrating Retrieval Augmented Generation (RAG). Tailored through prompt engineering and a new loss metric, Inverse Cosine Distance (ICD), designed for fine-tuning on large contextual datasets, LLMs efficiently generate structured meta-analysis content. Human evaluation then assesses relevance and provides information on model performance in key metrics. This research demonstrates that fine-tuned models outperform non-fine-tuned models, with fine-tuned LLMs generating 87.6% relevant meta-analysis abstracts. The relevance of the context, based on human evaluation, shows a reduction in irrelevancy from 4.56% to 1.9%. These experiments were conducted in a low-resource environment, highlighting the study's contribution to enhancing the efficiency and reliability of meta-analysis automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。