让大模型反向出题，发现它自造难题时的弱点。
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
- 用反向问答测试大模型生成难问题的能力
- 数值类答案在反向问答中准确率大幅下降
- 适合研究模型推理缺陷与评测基准设计
问答（QA）是常见任务，本文提出反向问答（RQA）：给定答案，生成对应问题。我们联合测试16个大模型在传统问答和反向问答上的表现，使用趣味知识数据集。结果表明：1）相比常规问答，大模型在数值类答案的反向问答中准确率显著下降，文本类答案则略有提升；2）模型常能正确回答自己生成的无效问题，说明错误不全是知识不足所致；3）反向问答错误与问题难度正相关，与答案在Dolma语料库中的出现频率负相关；4）模型难以生成有效的多跳问题。通过识别导致错误的问题与答案类型，我们为改进大模型推理提供建议。
原文摘要 · Abstract (English)
Question answering (QA), giving correct answers to questions, is a popular task, but we test reverse question answering (RQA): for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, aiding benchmark design, and checking reasoning consistency. We run 16 LLMs on QA and RQA with trivia questions/answers, revealing: 1) Versus QA, LLMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) LLMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not from knowledge gaps alone; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) LLMs struggle to provide valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。