让预测更可信:用论证一致性提升人类与大模型的预测准确率
Argumentatively Coherent Judgmental Forecasting
- 定义论证一致性:预测需与推理逻辑自洽
- 过滤不一致预测后,人与大模型准确率均提升
- 用户常忽视一致性,需自动筛选不靠谱意见
判断性预测依赖人类意见而非仅历史数据。当这些意见构成论证结构时,从论证角度研究预测特性具有价值。本文提出并形式化定义了论证一致性:预测者的推理必须与其预测结果逻辑自洽。通过三项评估验证该属性:首先,在人类与基于大语言模型(LLM)的预测者中,强制一致性显著提升预测准确率;其次,通过众包用户实验发现,尽管一致性看似合理且有效,但多数用户并未自发遵循该原则。这表明在基于论证的判断性预测中,需引入机制过滤不一致意见,以获得更可靠的群体预测结果。
原文摘要 · Abstract (English)
Judgmental forecasting employs human opinions to make predictions about future events, rather than exclusively historical data as in quantitative forecasting. When these opinions form an argumentative structure around forecasts, it is useful to study the properties of the forecasts from an argumentative perspective. In this paper, we advocate and formally define a property of argumentative coherence, which, in essence, requires that a forecaster's reasoning is coherent with their forecast. We then conduct three evaluations with our notion of coherence. First, we assess the impact of enforcing coherence on human forecasters as well as on Large Language Model (LLM)-based forecasters, given that they have recently shown to be competitive with human forecasters. In both cases, we show that filtering out incoherent predictions improves forecasting accuracy consistently, supporting the practical value of coherence in both human and LLM-based forecasting. Then, via crowd-sourced user experiments, we show that, despite its apparent intuitiveness and usefulness, users do not generally align with this coherence property. This points to the need to integrate, within argumentation-based judgmental forecasting, mechanisms to filter out incoherent opinions before obtaining group forecasting predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。