用大模型生成带常识的对话并自动评估,提升对话系统数据质量。
Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models
- 基于常识关系设计提示,让大模型生成多样化对话回复。
- 在5个数据集上生成200组对话,可识别12种特定常识关系。
- 无需复杂抽取,用指令提示实现自动评估,适合研究对话生成者。
本文探索了基于不同常识关系进行对话系统轮次级数据增强的初步结果,并提出了对生成合成对话轮次的自动评估方法。该方法利用预训练大语言模型(LLMs)的扩展知识与零样本能力,遵循指令、理解上下文信息及进行常识推理。思路借鉴链式思维(CoT),更明确地应用于基于常识属性的提示生成与自动评估。首先,从5个知名对话数据集中随机抽取200个不完整对话,生成受不同事件常识属性调控的替代回复,构建新数据集以衡量大模型生成上下文相关常识知识的能力,涵盖最多12种特定的ATOMIC [10]数据库关系。其次,提出一种自动评估框架,受ACCENT [26]指标启发,但不采用其复杂的事件-关系元组提取过程,而是为每种常识属性设计指令提示,使用先进大模型自动检测生成轮次中所用原始属性。初步结果显示,该方法有效利用大模型的常识推理与评估能力,适用于对话系统研究。
原文摘要 · Abstract (English)
This paper provides preliminary results on exploring the task of performing turn-level data augmentation for dialogue system based on different types of commonsense relationships, and the automatic evaluation of the generated synthetic turns. The proposed methodology takes advantage of the extended knowledge and zero-shot capabilities of pretrained Large Language Models (LLMs) to follow instructions, understand contextual information, and their commonsense reasoning capabilities. The approach draws inspiration from methodologies like Chain-of-Thought (CoT), applied more explicitly to the task of prompt-based generation for dialogue-based data augmentation conditioned on commonsense attributes, and the automatic evaluation of the generated dialogues. To assess the effectiveness of the proposed approach, first we extracted 200 randomly selected partial dialogues, from 5 different well-known dialogue datasets, and generate alternative responses conditioned on different event commonsense attributes. This novel dataset allows us to measure the proficiency of LLMs in generating contextually relevant commonsense knowledge, particularly up to 12 different specific ATOMIC [10] database relations. Secondly, we propose an evaluation framework to automatically detect the quality of the generated dataset inspired by the ACCENT [26] metric, which offers a nuanced approach to assess event commonsense. However, our method does not follow ACCENT's complex eventrelation tuple extraction process. Instead, we propose an instruction-based prompt for each commonsense attribute and use state-of-the-art LLMs to automatically detect the original attributes used when creating each augmented turn in the previous step. Preliminary results suggest that our approach effectively harnesses LLMs capabilities for commonsense reasoning and evaluation in dialogue systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。