用多模态数据预测网络安全演练成败，发现认知对齐比内容类型更重要。
Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?
- 通过学生邮件和任务日志，量化教学目标与实际行为的匹配度。
- 多模态特征组合预测准确率达AUC 0.80，远超仅用布鲁姆分类的模型。
- 认知对齐是关键诊断指标，适合教育技术与安全培训研究者参考。
教学对齐（instructional alignment）指预期认知与实际活动的一致性，是有效教学的核心但难于规模化评估。本研究基于23支团队（76名学生）在五次网络安全演练中的多模态数据，分析对齐程度。研究1利用布鲁姆分类法标注目标与团队邮件，通过广义线性混合模型建模关键任务完成情况。结果表明，所需与实际布鲁姆层级间的差异（即对齐度）能预测成功，而单独的布鲁姆类别则无法。研究2采用分组交叉验证与l1正则化逻辑回归比较不同特征族，文本嵌入与日志特征表现最佳（测试AUC约0.74和0.71），其组合效果最优（测试AUC约0.80），而布鲁姆频次贡献有限。整体而言，该研究提出一种模拟训练中的对齐度量方法，证明多模态数据最能预测表现，而对齐提供可解释的诊断洞察。
原文摘要 · Abstract (English)
Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationalize at scale. We examine alignment in cybersecurity simulations using multimodal traces from 23 teams (76 students) across five exercise sessions. Study 1 codes objectives and team emails with Bloom's taxonomy and models the completion of key exercise tasks with generalized linear mixed models. Alignment, defined as the discrepancy between required and enacted Bloom levels, predicts success, whereas the Bloom category alone does not predict success once discrepancy is considered. Study 2 compares predictive feature families using grouped cross-validation and l1-regularized logistic regression. Text embeddings and log features outperform Bloom-only models (AUC~0.74 and 0.71 vs. 0.55), and their combination performs best (Test AUC~0.80), with Bloom frequencies adding little. Overall, the work offers a measure of alignment for simulations and shows that multimodal traces best forecast performance, while alignment provides interpretable diagnostic insight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。