对比5种大模型在多种对话任务中的表现，发现无万能模型。
Strategic Prompting for Conversational Tasks: A Comparative Analysis of Large Language Models Across Diverse Conversational Tasks
- 选取Llama、OPT等5个主流模型，覆盖预约、心理咨询等5类对话任务。
- 不同模型在各任务中表现差异大，无统一最优解。
- 研究结果帮助开发者按任务选型，避免盲目使用单一模型。
随着对话人工智能的发展，评估大型语言模型（LLMs）在各类对话任务中的表现至关重要。本文全面评估了五种主流模型：Llama、OPT、Falcon、Alpaca和MPT，涵盖预约、共情回应生成、心理健康与法律咨询、说服及谈判等多样化任务。通过结合自动评估与人工评价的多维度测试框架，采用通用与任务特定指标进行综合衡量。结果显示，没有任何一个模型在所有任务中均表现最优，其性能随任务需求显著变化——某些模型在特定任务中表现优异，但在其他任务中则相对落后。该发现强调，在实际应用中必须根据具体任务特性选择合适的模型，而非依赖通用方案。
原文摘要 · Abstract (English)
Given the advancements in conversational artificial intelligence, the evaluation and assessment of Large Language Models (LLMs) play a crucial role in ensuring optimal performance across various conversational tasks. In this paper, we present a comprehensive study that thoroughly evaluates the capabilities and limitations of five prevalent LLMs: Llama, OPT, Falcon, Alpaca, and MPT. The study encompasses various conversational tasks, including reservation, empathetic response generation, mental health and legal counseling, persuasion, and negotiation. To conduct the evaluation, an extensive test setup is employed, utilizing multiple evaluation criteria that span from automatic to human evaluation. This includes using generic and task-specific metrics to gauge the LMs' performance accurately. From our evaluation, no single model emerges as universally optimal for all tasks. Instead, their performance varies significantly depending on the specific requirements of each task. While some models excel in certain tasks, they may demonstrate comparatively poorer performance in others. These findings emphasize the importance of considering task-specific requirements and characteristics when selecting the most suitable LM for conversational applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。