构建首个人工标注的提示数据集，提升大模型生成有效提示的能力。
WikiHint: A Human-Annotated Dataset for Hint Ranking and Generation
- 基于维基百科构建1000个问题的5000条人工提示数据
- 带答案信息的提示生成效果更优，编码器模型在提示排序中表现更好
- 适用于教育类AI、认知训练系统及可解释性研究
大型语言模型的广泛应用使得用户频繁向聊天机器人提问。在信息易得的时代，激发并保持人类的认知能力与推理技能变得尤为重要。本文通过推广提示作为直接答案的替代或补充，应对这一挑战。我们首先构建了一个手动标注的提示数据集WikiHint，该数据集基于维基百科，包含1000个问题和5000条提示。随后，我们在开源LLM上对提示生成任务进行微调，涵盖有答案信息和无答案信息两种场景。通过真人参与者对比使用提示与不使用提示的答题效果，评估提示有效性。此外，我们提出一种轻量级评估方法HintRank，用于在两种设置下对提示进行评估与排序。结果表明：(a) 该数据集有助于生成更有效的提示；(b) 在问题中加入答案信息通常能提升生成提示的质量；(c) 编码器模型在提示排序任务中表现优于解码器模型。
原文摘要 · Abstract (English)
The use of Large Language Models (LLMs) has increased significantly with users frequently asking questions to chatbots. In the time when information is readily accessible, it is crucial to stimulate and preserve human cognitive abilities and maintain strong reasoning skills. This paper addresses such challenges by promoting the use of hints as an alternative or a supplement to direct answers. We first introduce a manually constructed hint dataset, WikiHint, which is based on Wikipedia and includes 5,000 hints created for 1,000 questions. We then finetune open-source LLMs for hint generation in answer-aware and answer-agnostic contexts. We assess the effectiveness of the hints with human participants who answer questions with and without the aid of hints. Additionally, we introduce a lightweight evaluation method, HintRank, to evaluate and rank hints in both answer-aware and answer-agnostic settings. Our findings show that (a) the dataset helps generate more effective hints, (b) including answer information along with questions generally improves the quality of generated hints, and (c) encoder-based models perform better than decoder-based models in hint ranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。