arXiv:2502.00857cs.CLcs.IR2025-02中稿 · EMNLP被引 4

开源工具包统一提示生成与评估,助力AI引导式问答研究

HintEval: An Open-Source Python Toolkit for Hint Generation and Hint Evaluation

  • 提供标准化接口,支持多种提示生成方法
  • 集成多维度评估指标,实现跨数据集对比分析
  • 适合自然语言处理与信息检索领域研究者使用

大型语言模型直接回答用户问题,可能削弱批判性思维。提示生成通过不直接给出答案来引导用户思考,提示评估则衡量引导质量。当前研究受限于数据集分散、标注格式不一及工具专用化。为此,我们提出HintEval——一个开源Python工具包,统一管理多样提示数据集,支持基于答案和无答案的生成方法,并在统一数据模型中实现多种评估指标。该工具包支持可复现实验、跨数据集分析与多维评估,仅需少量工程工作。通过人机实验验证,参与者使用生成提示后正确解答率显著提升。附带完整文档、可运行的Google Colab笔记本及演示视频。通过规范评估流程与降低入门门槛,推动提示引导式问答在NLP与IR领域的系统性研究。

原文摘要 · Abstract (English)

Large Language Models (LLMs) increasingly provide direct answers to user questions, raising concerns about reduced engagement in critical thinking and problem-solving. Hint generation offers an alternative by guiding users toward answers without revealing them, while hint evaluation assesses the quality of such guidance. Research in this area is hindered by fragmented datasets, inconsistent annotation formats, and evaluation tools that are often dataset-specific or unavailable. To address these challenges, we introduce HintEval, an open-source Python library for unified hint generation and evaluation. HintEval standardizes access to diverse hint datasets, supports answer-aware and answer-agnostic generation methods, and implements multiple evaluation metrics within a shared data model. The toolkit enables reproducible experimentation, cross-dataset analysis, and multi-dimensional evaluation with minimal engineering effort. We further demonstrate its utility through human studies in which participants assess generated hints and use them to answer questions, showing that hints can effectively support users in reaching correct answers. HintEval is accompanied by comprehensive documentation, an executable Google Colab notebook for rapid experimentation, and a demonstration video. By promoting consistent evaluation practices and lowering barriers to entry, it facilitates systematic research on hint-based question answering (QA) in NLP and IR.

提示生成评估工具自然语言处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。