用维基百科生成73万条长对话，提升大模型连贯对话能力。
DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities
- 从文本集合构建多轮多主题问答对话，模拟真实交流场景。
- 预训练时加入该数据，上下文理解能力提升40%且基础性能不变。
- 适合想提升模型对话连贯性与记忆能力的研究者使用。
大型语言模型在多轮对话任务中应用日益广泛，但其预训练数据主要为连续段落文本，与实际对话需求存在差距。本文提出一种新方法，从现有文本语料中合成对话数据。构建了一套将一组相关文档转化为扩展型多轮、多主题信息查询对话的流程。基于维基百科文章，我们构建了包含超过73万条长对话的DocTalk多轮预训练对话语料库。假设在预训练阶段引入此类合成对话结构，可增强大模型的基本多轮对话能力，如上下文记忆与理解。实验表明，在预训练中使用DocTalk可使上下文记忆与理解能力提升最高达40%，且不损害基础性能。DocTalk已公开于https://huggingface.co/datasets/AmazonScience/DocTalk。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly employed in multi-turn conversational tasks, yet their pre-training data predominantly consists of continuous prose, creating a potential mismatch between required capabilities and training paradigms. We introduce a novel approach to address this discrepancy by synthesizing conversational data from existing text corpora. We present a pipeline that transforms a cluster of multiple related documents into an extended multi-turn, multi-topic information-seeking dialogue. Applying our pipeline to Wikipedia articles, we curate DocTalk, a multi-turn pre-training dialogue corpus consisting of over 730k long conversations. We hypothesize that exposure to such synthesized conversational structures during pre-training can enhance the fundamental multi-turn capabilities of LLMs, such as context memory and understanding. Empirically, we show that incorporating DocTalk during pre-training results in up to 40% gain in context memory and understanding, without compromising base performance. DocTalk is available at https://huggingface.co/datasets/AmazonScience/DocTalk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。