用360度视频+AI自动识别动作,提升关键训练复盘效率。
ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefing
- 基于360度视频与空间注意力机制,精准检测动作。
- 模型压缩74%后仍保持高精度,推理速度显著提升。
- 集成大模型生成总结报告,适合军事、应急等场景。
在灾难响应、军事模拟和工业安全等高风险任务中,高效训练与复盘至关重要。传统依赖人工观看2D视频的分析方式耗时且缺乏全局视角。为此,我们提出ACT360系统,利用360度视频与机器学习实现自动化动作检测与结构化复盘。该系统融合360YOWO模型,通过空间注意力与等距柱状投影感知卷积(EAC)缓解全景视频畸变。为适配资源受限环境,采用量化与剪枝技术,使模型体积缩小74%,仅导致平均精度(mAP)下降1.5%(从0.865降至0.850),同时提升推理速度。我们在包含55段标注360度视频的公开数据集上验证方法,涵盖七类关键操作,覆盖多种真实训练场景与环境条件。ACT360还集成360AIE(Action Insight Explorer)——一个基于大语言模型的网页界面,支持自动动作检测、检索与文本摘要,大幅提升事后分析效率。该框架具备通用性,适用于需轻量级动作检测与结构化分析的各类训练场景。
原文摘要 · Abstract (English)
Effective training and debriefing are critical in high-stakes, mission-critical environments such as disaster response, military simulations, and industrial safety, where precision and minimizing errors are paramount. The traditional post-training analysis relies on manually reviewing 2D videos, a time-consuming process that lacks comprehensive situational awareness. To address these limitations, we introduce ACT360, a system that leverages 360-degree videos and machine learning for automated action detection and structured debriefing. ACT360 integrates 360YOWO, an enhanced You Only Watch Once (YOWO) model with spatial attention and equirectangular-aware convolution (EAC) to mitigate panoramic video distortions. To enable deployment in resource-constrained environments, we apply quantization and model pruning, reducing the model size by 74% while maintaining robust accuracy (mAP drop of only 1.5%, from 0.865 to 0.850) and improving inference speed. We validate our approach on a publicly available dataset of 55 labeled 360-degree videos covering seven key operational actions, recorded across various real-world training sessions and environmental conditions. Additionally, ACT360 integrates 360AIE (Action Insight Explorer), a web-based interface for automatic action detection, retrieval, and textual summarization using large language models (LLMs), significantly enhancing post-incident analysis efficiency. ACT360 serves as a generalized framework for mission-critical debriefing, incorporating EAC, spatial attention, summarization, and model optimization. These innovations apply to any training environment requiring lightweight action detection and structured post-exercise analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。