用大模型自动采集事故新闻，生成可靠数据集。
Design and Application of Multimodal Large Language Model Based System for End to End Automation of Accident Dataset Generation
- 用多模态大模型自动生成网页爬虫代码并分类抓取新闻。
- 111天内处理超1.5万篇新闻，提取705起真实交通事故数据。
- 适合交通研究者、政策制定者快速获取高精度事故数据。
道路交通事故仍是孟加拉国等发展中国家的重大公共安全与社会经济问题。现有事故数据采集主要依赖人工，分散且不可靠，导致报告不足和记录不一。本研究提出一个基于大语言模型（LLMs）和网络爬虫技术的全流程自动化系统，解决上述挑战。该流程包含四个模块：自动化爬虫代码生成、在线新闻收集、事故新闻分类与结构化信息提取、去重处理。系统采用多模态生成式大模型Gemini-2.0-Flash实现无缝自动化。代码生成模块将网页分为分页、动态加载或无限滚动三类，并生成相应Python脚本。大模型还负责分类并提取关键事故信息，如日期、时间、地点、死亡人数、受伤人数、道路类型、车辆类型及行人参与情况。去重算法保障数据完整性。系统在2024年10月1日至2025年1月20日间，对孟加拉国14个主要新闻网站进行爬取，处理超过15,000篇新闻文章，识别出705起独特事故事件。代码生成模块达到91.3%校准率和80%验证准确率。吉大港事故数量最多（80起），死亡70人，受伤115人，其次为达卡、法里德布尔、加齐布尔和考克斯巴扎尔。事故高峰集中在早间（8-9点）、中午（12-1点）和晚间（6-7点）。同时建立了公开数据仓库并附使用说明。本研究展示了基于大模型的可扩展、低投入、高精度事故数据采集系统的可行性，为孟加拉国数据驱动的道路安全政策制定奠定基础。
原文摘要 · Abstract (English)
Road traffic accidents remain a major public safety and socio-economic issue in developing countries like Bangladesh. Existing accident data collection is largely manual, fragmented, and unreliable, resulting in underreporting and inconsistent records. This research proposes a fully automated system using Large Language Models (LLMs) and web scraping techniques to address these challenges. The pipeline consists of four components: automated web scraping code generation, news collection from online sources, accident news classification with structured data extraction, and duplicate removal. The system uses the multimodal generative LLM Gemini-2.0-Flash for seamless automation. The code generation module classifies webpages into pagination, dynamic, or infinite scrolling categories and generates suitable Python scripts for scraping. LLMs also classify and extract key accident information such as date, time, location, fatalities, injuries, road type, vehicle types, and pedestrian involvement. A deduplication algorithm ensures data integrity by removing duplicate reports. The system scraped 14 major Bangladeshi news sites over 111 days (Oct 1, 2024 - Jan 20, 2025), processing over 15,000 news articles and identifying 705 unique accidents. The code generation module achieved 91.3% calibration and 80% validation accuracy. Chittagong reported the highest number of accidents (80), fatalities (70), and injuries (115), followed by Dhaka, Faridpur, Gazipur, and Cox's Bazar. Peak accident times were morning (8-9 AM), noon (12-1 PM), and evening (6-7 PM). A public repository was also developed with usage instructions. This study demonstrates the viability of an LLM-powered, scalable system for accurate, low-effort accident data collection, providing a foundation for data-driven road safety policymaking in Bangladesh.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。