跨数据源枪击风险分类存在严重失效，主因是特征不完整而非模型选择。
MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification
- 构建6968起事件的标准化基准MASH-Bench，评估跨源分类性能。
- 在GVA数据上召回率仅0.20，精度0.0004，特征缺失导致召回归零。
- 特征完整性比模型选择更重要，适合关注数据质量与公平性的研究者。
公共大规模枪击数据库在覆盖范围、特征可用性和报告实践上差异显著，给需跨数据源泛化的机器学习模型带来挑战。本文提出MASH-Bench，一个包含来自四个美国数据库（Kaggle、Mother Jones、Stanford MSA、Gun Violence Archive，简称GVA）共6,968起事件的标准化基准。通过留一数据集外（LODO）评估跨源风险分类性能。随机森林、XGBoost和LightGBM在已校准数据源上的高风险召回率达0.68–0.89，但在GVA上平均召回降至0.20，精度低至0.0004。通过受控特征掩码消融实验，移除在GVA中缺失的五个特征后，召回率归零，表明特征完整性是跨源失败的关键因素。进一步评估三种领域自适应方法：DANN使GVA上高风险召回提升0.282（95%置信区间[0.11, 0.47]，p=0.003），但精度仍低；CORAL与重要性加权均未提升召回。最优先验偏移校准也未能恢复高风险预测，说明仅修正标签不足。群体审计发现媒体标注的心理健康标签存在显著偏差。结果表明，在MASH-Bench中，跨源泛化受限于特征完整性与标签分布，而非分类器选择。该基准为诊断此类问题提供可控环境。
原文摘要 · Abstract (English)
Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。