构建首个面向个性化助手的开源数据集与评估工具。
Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis
- 提出HiCUPID数据集与基于Llama-3.2的自动化评估模型。
- 评估结果与人类偏好高度一致，验证了模型有效性。
- 适合研究个性化对话、LLM评估与应用的开发者使用。
个性化AI助手是大语言模型（LLMs）实现类人能力的重要体现，但其发展受限于缺乏专门用于个性化的开源对话数据集。为填补这一空白，本文提出HiCUPID基准，包含一个对话数据集及基于Llama-3.2的自动化评估模型，该模型评估结果与人类偏好高度一致。我们公开发布数据集、评估模型与代码，链接为https://github.com/12kimih/HiCUPID，旨在推动个性化助手的研究进展。
原文摘要 · Abstract (English)
Personalized AI assistants, a hallmark of the human-like capabilities of Large Language Models (LLMs), are a challenging application that intertwines multiple problems in LLM research. Despite the growing interest in the development of personalized assistants, the lack of an open-source conversational dataset tailored for personalization remains a significant obstacle for researchers in the field. To address this research gap, we introduce HiCUPID, a new benchmark to probe and unleash the potential of LLMs to deliver personalized responses. Alongside a conversational dataset, HiCUPID provides a Llama-3.2-based automated evaluation model whose assessment closely mirrors human preferences. We release our dataset, evaluation model, and code at https://github.com/12kimih/HiCUPID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。