首个大规模危机预警生成数据集,助力AI实时生成精准应急提醒。
CrisiText: A dataset of warning messages for LLM training in emergency communication
- 基于13类灾情构建40万条预警消息链,每条含真实场景与专家指导术语
- 实验验证偏好对齐优于监督微调,且在分布外场景中表现更稳
- 适合应急系统研发、灾害通信与NLG模型评估的研究者使用
在自然灾害或暴力袭击等危机情境中,及时识别威胁并减少损失至关重要。尽管人工智能已在辅助应急方面发挥作用,但自然语言处理技术仍局限于分类任务,而基于自然语言生成(NLG)的实时预警潜力尚未被充分挖掘。本文提出CrisiText,首个涵盖13类危机场景的大规模预警消息生成数据集,包含超过40万条预警信息(覆盖近18,000个具体事件),旨在帮助民众在危机发生前后获得有效指引。数据通过现有危机描述构建事件链,每个事件匹配一条符合专家规范的警告消息,并附带三种劣化版本以支持不同NLG方法研究。我们对比了监督微调、偏好对齐、零样本及少样本等多种方法,进一步评估了模型在分布外场景的表现,并测试了一种自动后编辑器的有效性。
原文摘要 · Abstract (English)
Effectively identifying threats and mitigating their potential damage during crisis situations, such as natural disasters or violent attacks, is paramount for safeguarding endangered individuals. To tackle these challenges, AI has been used in assisting humans in emergency situations. Still, the use of NLP techniques remains limited and mostly focuses on classification tasks. The significant potential of timely warning message generation using NLG architectures, however, has been largely overlooked. In this paper we present CrisiText, the first large-scale dataset for the generation of warning messages across 13 different types of crisis scenarios. The dataset contains more than 400,000 warning messages (spanning almost 18,000 crisis situations) aimed at assisting civilians during and after such events. To generate the dataset, we started from existing crisis descriptions and created chains of events related to the scenarios. Each event was then paired with a warning message. The generations follow experts' written guidelines to ensure correct terminology and factuality of their suggestions. Additionally, each message is accompanied by three suboptimal warning types to allow for the study of different NLG approaches. To this end, we conducted a series of experiments comparing supervised fine-tuning setups with preference alignment, zero-shot, and few-shot approaches. We further assessed model performance in out-of-distribution scenarios and evaluated the effectiveness of an automatic post-editor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。