arXiv:2601.18271cs.DLcs.CL2026-01

用1446条乱序文本挑战LLM精准提取科研评分,推动提示工程进步。

Designing large language model prompts to extract scores from messy text: A shared dataset and challenge

  • 设计提示让LLM从杂乱文本中准确提取1到4星评分。
  • 初始方案准确率72.6%,需突破该基准。
  • 适合研究提示工程与信息抽取的开发者和学者。

在计算、自然语言处理与信息科学领域,通过共享数据集并发起挑战,推动算法进步。本文发布一个包含1446条短文本的共享数据集,每条文本描述英国科研质量评分(1*至4*)。这些文本存在缺失评分、无效值或异常格式等问题。数据集提供有效评分标准及“黄金标准”答案(含缺失值标记-1)。挑战目标是设计一个大语言模型(LLM)提示,使其仅输出数字(或-1表示缺失),实现最高准确率。提示需明确指导模型何时返回-1。文章还提供一个基础提示示例。本挑战旨在解决实际问题,同时深化对提示设计与LLM处理复杂数值任务能力的理解。初始方法准确率为72.6%,目标是超越此水平。

原文摘要 · Abstract (English)

In some areas of computing, natural language processing and information science, progress is made by sharing datasets and challenging the community to design the best algorithm for an associated task. This article introduces a shared dataset of 1446 short texts, each of which describes a research quality score on the UK scale of 1* to 4*. This is a messy collection, with some texts not containing scores and others including invalid scores or strange formats. With this dataset there is also a description of what constitutes a valid score and a "gold standard" of the correct scores for these texts (including missing values). The challenge is to design a prompt for Large Language Models (LLMs) to extract the scores from these texts as accurately as possible. The format for the response should be a number and no other text so there are two aspects to the challenge: ensuring that the LLM returns only a number, and instructing it to deduce the correct number for the text. As part of this, the LLM prompt needs to explain when to return the missing value code, -1, instead of a number when the text does not clearly contain one. The article also provides an example of a simple prompt. The purpose of the challenge is twofold: to get an effective solution to this problem, and to increase understanding of prompt design and LLM capabilities for complex numerical tasks. The initial solution suggested has an accuracy of 72.6%, so the challenge is to beat this.

提示工程信息抽取大模型评分提取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。