测试大模型对短问题的事实性回答能力，判断其是否知道自己的知识边界。
Measuring short-form factuality in large language models
- 通过对抗性收集数据，让模型难以靠推测答对
- 每题仅有一个明确正确答案，便于客观评分
- 适合评估模型在不确定时是否拒绝作答
我们提出SimpleQA，一个评估语言模型回答简短事实性问题能力的基准。设计时优先考虑两点：一是挑战性，数据集针对GPT-4的回答进行对抗性收集；二是可评分性，问题设计确保仅有唯一明确答案。每个回答被标记为正确、错误或未尝试。理想模型应尽可能答对已知问题，对不确定的问题则不作答。SimpleQA旨在衡量模型是否“知道自己知道什么”，希望该基准能持续适用于未来数代前沿模型。项目地址：https://github.com/openai/simple-evals。
原文摘要 · Abstract (English)
We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。