arXiv:2608.04009cs.CL2026-08被引 1

评测大模型预测社会事件发展能力,发现其表现远未达标。

SocietyBench: Forecasting Counterfactual Social-World Evolution

论文配图:SocietyBench: Forecasting Counterfactual Social-World Evolution
图 1 · 摘自论文原文
  • 构建反事实社会事件时间线,避免模型依赖记忆匹配
  • 六种前沿模型最高仅达75.0分(满分100),远超随机基线
  • 提出双维度评估:概率校准与时间准确性,两者可分离

大型语言模型及其代理系统当前主要通过任务完成度(如修复漏洞、操作界面)进行评测,而对真实社会事件演变的理解与预测能力却几乎未被衡量。本文提出SocietyBench,一个端到端基准测试框架:输入事件主题后,从五个平台抓取新闻与社交媒体内容,构建包含事实事件与公众意见两层的日期索引时间线,并将每个时间点转化为可审计的预测题库。题目在两个独立的100分维度上评分:概率校准性与时间准确性。在模型接触时间线前,通过三阶段流程替换所有命名实体并统一偏移日期,生成结构相同但表面标签被抹除的反事实社会世界,以防止模型利用预训练记忆。在中英文双语下涵盖五个异构事件、125个预测点,六种前沿模型最高得分仅75.0,显著高于随机基线50。两个维度可分离:模型可在校准性上强但时间判断弱,反之亦然。三种基于同一基础模型的代理框架未能超越基线,两种无模型启发式方法也落后于所有大模型。单事件最大差距达21.4分,支持多事件综合评估。所有匿名化时间线、题库、真实答案及评分代码均已公开。

原文摘要 · Abstract (English)

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

社会推理反事实预测评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。