用NLP技术分析航空事故报告,找出隐藏的主题模式。
Exploring Aviation Incident Narratives Using Topic Modeling and Clustering Techniques
- 采用LDA等5种主题模型分析事故文本,挖掘潜在主题
- LDA表现最佳,主题一致性达0.597,优于其他模型
- 适合安全研究者和民航数据分析人员参考
航空安全是全球性议题,需深入分析事故以全面理解影响因素。本研究使用美国国家运输安全委员会(NTSB)数据集,应用隐含狄利克雷分配(LDA)、非负矩阵分解(NMF)、潜在语义分析(LSA)、概率潜在语义分析(pLSA)及K-means聚类等自然语言处理技术,旨在识别隐含主题、探索语义关联、评估概率联系,并根据共性特征对事故进行聚类。结果表明,LDA在主题一致性上表现最优(0.597),其次为pLSA(0.583)、LSA(0.542)和NMF(0.437)。K-means聚类进一步揭示了事故叙述中的共性和独特洞察。研究揭示了事故叙述中的潜在模式与主题结构,提供了多种主题建模方法的对比分析。未来可探索时间趋势、整合更多数据集,并构建早期安全风险预测模型。该研究为深化航空安全理解与改进奠定了基础。
原文摘要 · Abstract (English)
Aviation safety is a global concern, requiring detailed investigations into incidents to understand contributing factors comprehensively. This study uses the National Transportation Safety Board (NTSB) dataset. It applies advanced natural language processing (NLP) techniques, including Latent Dirichlet Allocation (LDA), Non-Negative Matrix Factorization (NMF), Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (pLSA), and K-means clustering. The main objectives are identifying latent themes, exploring semantic relationships, assessing probabilistic connections, and cluster incidents based on shared characteristics. This research contributes to aviation safety by providing insights into incident narratives and demonstrating the versatility of NLP and topic modelling techniques in extracting valuable information from complex datasets. The results, including topics identified from various techniques, provide an understanding of recurring themes. Comparative analysis reveals that LDA performed best with a coherence value of 0.597, pLSA of 0.583, LSA of 0.542, and NMF of 0.437. K-means clustering further reveals commonalities and unique insights into incident narratives. In conclusion, this study uncovers latent patterns and thematic structures within incident narratives, offering a comparative analysis of multiple-topic modelling techniques. Future research avenues include exploring temporal patterns, incorporating additional datasets, and developing predictive models for early identification of safety issues. This research lays the groundwork for enhancing the understanding and improvement of aviation safety by utilising the wealth of information embedded in incident narratives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。