构建951条概念性论证评分数据集，评估大模型在无标准答案问题上的推理能力。
A dataset of rated conceptual arguments
- 基于论证可评性，构建涵盖AI安全等领域的批判性观点数据集
- 6位专家对1458条论证打分，涵盖中心性、强度、正确性等维度
- 验证大模型表现与通用能力排名一致，适合研究可信推理的学者
大语言模型在数学和编程等有明确答案的任务上进步迅速，但在概念性问题上的推理能力仍不清晰。这类问题无客观答案也无公认解决方法，但可通过辩论推进。我们提出：尽管结论难评，具体论证却可更可靠地评估。为此，我们构建了一个包含951条论证批判的语料库，覆盖442个立场文本，主题涉及AI安全、决策理论、伦理与政治，由6位专家从中心性、强度、正确性和清晰度等维度进行1458次评分。我们设计两种评分函数并测试多种模型，结果表明模型表现与通用能力排名高度一致。
原文摘要 · Abstract (English)
Large language models have improved rapidly on tasks with verifiable answers, such as mathematics and programming. Much less is known about their ability to reason about what we call conceptual questions: questions for which no ground truth is realistically accessible and no widely accepted resolution methodology exists, but on which progress can still be made by debating arguments. Most philosophical questions are of this kind, as are central components of questions in AI safety, decision theory, and social choice. Our approach is based on the view that while bottom-line conclusions on such questions are hard to evaluate, individual contextualized arguments can be evaluated far more reliably. We therefore introduce a dataset of 951 argumentative critiques of 442 position texts, spanning topics from AI safety and decision theory to ethics and politics, with 1,458 ratings by six expert raters along dimensions including centrality, strength, correctness, and clarity. We propose two scoring functions and benchmark a range of models. Performance tracks general capability rankings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。