用新闻数据更新大模型知识，准确率从2%提至82%
PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs
- 通过改写+自生成问答增强数据，提升知识注入效果
- 自学习强化过程使准确率从0.02升至0.82，且不损失通用能力
- 适合需要精准事实回答的垂直领域模型构建
预训练大语言模型的知识更新仍是重要挑战。尽管持续训练提供了潜在路径，但仍存在显著技术难题。此外，大模型在回答特定事实类问题（如新闻文章）方面表现不佳，这一局限广受学界认可。本文提出PASTA框架，通过整合新闻文章中的详细事实信息，实现对大模型的知识注入，旨在构建能准确回答相关问题的专用模型。该框架结合数据增强、问答生成与一种新型自学习DPO机制，同时实现知识覆盖与幻觉抑制。通过系统分析学习参数与数据配置，揭示有效更新策略。在基线模型知识截止日期后发布的网页文章上评估，PASTA将准确率从0.02提升至0.82，同时保持通用语言能力，证明其在构建领域专用大模型方面的有效性。
原文摘要 · Abstract (English)
Knowledge updating in pre-trained Large Language Models (LLMs) remains an important challenge. While continual training provides a potential avenue for knowledge updating, it continues to present substantial technical difficulties. Furthermore, LLMs often struggle with accurately answering questions about specific factual information, such as news articles - a capability limitation widely recognized in the research community. This paper proposes PASTA, a simple yet powerful framework for integrating detailed factual information from news articles as new knowledge into LLMs, with the primary goal of building specialized models that accurately answer questions about this knowledge. Our framework combines data augmentation, question-answering generation, and a novel self-learning DPO process that simultaneously enables knowledge overwriting and hallucination suppression. We provide insights into effective knowledge updating through systematic analysis of learning parameters and data configurations. In our experimental evaluation with web articles published after the base model's knowledge cutoff, PASTA achieved remarkable improvement from 0.02 to 0.82 accuracy while maintaining general language capabilities, demonstrating its effectiveness for creating domain-specialized LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。