用自生成反馈训练大模型，让对话问答更准
Preference-based Learning with Retrieval Augmented Generation for Conversational Question Answering
- 分三步训练：理解问题、找证据、生成答案，每步用模型自生成数据
- 在基准测试上准确率比基线提升15.5个百分点，达到新最好水平
- 适合做对话问答系统研发，尤其缺乏标注数据的场景
对话问答（ConvQA）包含多个子任务：理解上下文中的不完整问题、检索相关信息、生成答案。本文提出PRAISE，一种基于流水线的ConvQA方法，为每个子任务训练大型语言模型适配器。由于实际中缺乏各子任务的标注数据，PRAISE通过自身生成结果，以最终回答性能作为反馈信号进行无监督训练，并将中间信息（如相关证据）视为弱标签数据。针对每个子任务，采用直接偏好优化，对比成功与失败样本进行学习。实验表明，该训练范式有效：PRAISE在各子任务上均表现提升，在主流ConvQA基准上取得新最优性能，相比基线精度提高15.5个百分点。
原文摘要 · Abstract (English)
Conversational Question Answering (ConvQA) involves multiple subtasks, i) to understand incomplete questions in their context, ii) to retrieve relevant information, and iii) to generate answers. This work presents PRAISE, a pipeline-based approach for ConvQA that trains LLM adapters for each of the three subtasks. As labeled training data for individual subtasks is unavailable in practice, PRAISE learns from its own generations using the final answering performance as feedback signal without human intervention and treats intermediate information, like relevant evidence, as weakly labeled data. We apply Direct Preference Optimization by contrasting successful and unsuccessful samples for each subtask. In our experiments, we show the effectiveness of this training paradigm: PRAISE shows improvements per subtask and achieves new state-of-the-art performance on a popular ConvQA benchmark, by gaining 15.5 percentage points increase in precision over baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。