用多人评审机制生成更复杂多轮对话，提升大模型对话能力。
ReviewInstruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models
- 通过提问-回应-评审三角色迭代优化指令生成
- 在MT-Bench和MMLU-Pro上分别提升2%和2.9%性能
- 适合需要高质量对话数据的AI研发人员
大语言模型在对话AI中的表现受限于单轮监督微调数据，导致多轮对话上下文连贯性差。现有方法难以兼顾指令的多样性和质量。为此，我们提出Review-Instruct框架，通过包含候选者、多名评审员和主席三个角色的“提问-回应-评审”迭代过程，合成多轮对话。该框架通过引入评审反馈不断优化指令，提升对话多样性与难度。我们基于Alpaca数据集构建多轮对话数据集，并对LLaMA2-13B模型进行微调。在MT-Bench、MMLU-Pro和Auto-Arena上的评估显示，相比基于LLaMA2-13B的先进模型，取得2.9%（MMLU-Pro）和2%（MT-Bench）的绝对提升。消融实验证实评审阶段及多评审员设计对提升指令多样性和难度至关重要。本工作展示了评审驱动、多智能体框架在大规模生成高质量对话数据方面的潜力。
原文摘要 · Abstract (English)
The effectiveness of large language models (LLMs) in conversational AI is hindered by their reliance on single-turn supervised fine-tuning (SFT) data, which limits contextual coherence in multi-turn dialogues. Existing methods for generating multi-turn dialogue data struggle to ensure both diversity and quality in instructions. To address this, we propose Review-Instruct, a novel framework that synthesizes multi-turn conversations through an iterative "Ask-Respond-Review" process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman. The framework iteratively refines instructions by incorporating Reviewer feedback, enhancing dialogue diversity and difficulty. We construct a multi-turn dataset using the Alpaca dataset and fine-tune the LLaMA2-13B model. Evaluations on MT-Bench, MMLU-Pro, and Auto-Arena demonstrate significant improvements, achieving absolute gains of 2.9\% on MMLU-Pro and 2\% on MT-Bench compared to prior state-of-the-art models based on LLaMA2-13B. Ablation studies confirm the critical role of the Review stage and the use of multiple Reviewers in boosting instruction diversity and difficulty. Our work highlights the potential of review-driven, multi-agent frameworks for generating high-quality conversational data at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。