用真实人类特征优化对话生成，让聊天机器人测评更像真人互动。
DiverseDialogue: A Methodology for Designing Chatbots with Human-Like Diversity
- 从真实人类对话中提取年龄、性别、情绪等特征，自动生成仿真用户提示。
- 使LLM生成对话的平均语言特征误差降低54%，显著提升多样性。
- 适合需要高真实感测评的教育、客服类聊天机器人开发者。
大型语言模型（LLMs）常被用于模拟人类用户以评估聊天机器人，如教学与客户服务场景。有效评估需模拟出高度类人化的多样性。本文表明，使用GPT-4o mini生成的对话在话题变化、词汇属性及语言使用平均行为与变异程度等多个语言特征上，系统性地不同于真实人类对话。为解决此差异，我们提出一种方法：通过引入来自真实人类互动的特征（如年龄、性别、情感基调、讨论话题）自动构建用户模拟提示。我们结合差分语言分析与深度语言探究评估该方法。针对特定语言特征定制的提示优化策略展现出显著改进，提升了聊天机器人对话的语言多样性。平均而言，人类与LLM生成对话间平均特征的误差降低了54%。该构建具备类人多样性的聊天机器人评测集的方法，极大增强了面向用户的机器人评估效能。
原文摘要 · Abstract (English)
Large Language Models (LLMs), which simulate human users, are frequently employed to evaluate chatbots in applications such as tutoring and customer service. Effective evaluation necessitates a high degree of human-like diversity within these simulations. In this paper, we demonstrate that conversations generated by GPT-4o mini, when used as simulated human participants, systematically differ from those between actual humans across multiple linguistic features. These features include topic variation, lexical attributes, and both the average behavior and diversity (variance) of the language used. To address these discrepancies, we propose an approach that automatically generates prompts for user simulations by incorporating features derived from real human interactions, such as age, gender, emotional tone, and the topics discussed. We assess our approach using differential language analysis combined with deep linguistic inquiry. Our method of prompt optimization, tailored to target specific linguistic features, shows significant improvements. Specifically, it enhances the human-likeness of LLM chatbot conversations, increasing their linguistic diversity. On average, we observe a 54 percent reduction in the error of average features between human and LLM-generated conversations. This method of constructing chatbot sets with human-like diversity holds great potential for enhancing the evaluation process of user-facing bots.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。