用化学领域数据预训练大模型，显著提升专业任务表现。
Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry
- 在化学领域数据上继续预训练基础模型
- 零样本下即达较好效果，指令微调后性能大幅提升
- 适合需要高准确性的化学文本生成与识别场景
大规模语言模型（如GPT系列、BLOOM、LLaMA等）在自然语言处理任务中表现出色，但在小众或专业领域易出现幻觉或错误输出。本文研究了在化学领域进行领域预训练对生成式大模型的益处，并对比了开源通用模型在零样本和少样本提示下的表现。结果表明，经过化学领域预训练的基线模型在零样本设置下即可在化学相关任务上表现良好，通过指令微调后，在命名实体识别和分子式生成等特定任务上取得显著进步。
原文摘要 · Abstract (English)
A proliferation of Large Language Models (the GPT series, BLOOM, LLaMA, and more) are driving forward novel development of multipurpose AI for a variety of tasks, particularly natural language processing (NLP) tasks. These models demonstrate strong performance on a range of tasks; however, there has been evidence of brittleness when applied to more niche or narrow domains where hallucinations or fluent but incorrect responses reduce performance. Given the complex nature of scientific domains, it is prudent to investigate the trade-offs of leveraging off-the-shelf versus more targeted foundation models for scientific domains. In this work, we examine the benefits of in-domain pre-training for a given scientific domain, chemistry, and compare these to open-source, off-the-shelf models with zero-shot and few-shot prompting. Our results show that not only do in-domain base models perform reasonably well on in-domain tasks in a zero-shot setting but that further adaptation using instruction fine-tuning yields impressive performance on chemistry-specific tasks such as named entity recognition and molecular formula generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。