用大模型模拟专家风险评估，效率提升百倍。
Scalable Delphi: Large Language Models for Structured Risk Estimation
- 设计多角色大模型迭代评估流程，模仿传统德尔菲法
- 与基准数据相关性达0.87~0.95，随证据增加持续优化
- 适合急需快速风险评估的高危场景，如AI安全
高风险领域中的量化风险评估依赖结构化专家征询来估算不可观测属性。传统黄金标准——德尔菲法——虽能生成校准且可审计的判断，但需数月协调与专家投入，使严谨评估难以普及。本文探索大语言模型（LLMs）是否可作为结构化专家征询的可扩展替代方案。提出可扩展德尔菲（Scalable Delphi），将经典协议适配至大模型，引入多样化专家角色、迭代修正与推理共享机制。由于目标量通常不可观测，构建基于必要条件的评估框架：对可验证代理的校准性、对证据的敏感性，以及与人类专家判断的一致性。在人工智能增强型网络安全风险领域进行评估，使用三个能力基准和独立的人类征询研究。结果显示，大模型小组与基准真实值相关性达皮尔逊系数0.87–0.95，随着证据增加系统性改进，并与人类专家小组高度一致——在一次对比中，比两个真人小组彼此之间的差异更接近。表明基于大模型的征询可将结构化专家判断拓展至传统方法不可行的场景，将征询时间从数月压缩至数分钟。
原文摘要 · Abstract (English)
Quantitative risk assessment in high-stakes domains relies on structured expert elicitation to estimate unobservable properties. The gold standard - the Delphi method - produces calibrated, auditable judgments but requires months of coordination and specialist time, placing rigorous risk assessment out of reach for most applications. We investigate whether Large Language Models (LLMs) can serve as scalable proxies for structured expert elicitation. We propose Scalable Delphi, adapting the classical protocol for LLMs with diverse expert personas, iterative refinement, and rationale sharing. Because target quantities are typically unobservable, we develop an evaluation framework based on necessary conditions: calibration against verifiable proxies, sensitivity to evidence, and alignment with human expert judgment. We evaluate in the domain of AI-augmented cybersecurity risk, using three capability benchmarks and independent human elicitation studies. LLM panels achieve strong correlations with benchmark ground truth (Pearson r=0.87-0.95), improve systematically as evidence is added, and align with human expert panels - in one comparison, closer to a human panel than the two human panels are to each other. This demonstrates that LLM-based elicitation can extend structured expert judgment to settings where traditional methods are infeasible, reducing elicitation time from months to minutes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。