针对罕见模糊错误，用双阶段专家模型提升视频误判检测效果
Dual-Stage Reweighted MoE for Long-Tailed Egocentric Mistake Detection
- 双阶段设计：特征级与分类级专家融合，结合冻结与微调模型
- 三种损失联合优化，显著提升对稀有错误的识别率与模型校准性
- 适合需要高精度检测少见操作失误的场景，如教学指导与安全监控
本文研究从第一人称视频中判断用户动作是否出错的问题。针对细微且罕见的错误难以检测的挑战，提出双阶段重加权混合专家（DR-MoE）框架。第一阶段使用冻结的ViViT与LoRA微调的ViViT提取特征，并通过特征级专家模块融合；第二阶段训练三个具有不同目标的分类器：重加权交叉熵缓解类别不平衡，AUC损失提升分布偏斜下的排序性能，标签感知损失结合锐度感知最小化增强校准性与泛化能力。三者预测通过分类级专家模块融合。该方法在识别罕见和模糊错误实例上表现优异。代码已开源：https://github.com/boyuh/DR-MoE。
原文摘要 · Abstract (English)
In this report, we address the problem of determining whether a user performs an action incorrectly from egocentric video data. To handle the challenges posed by subtle and infrequent mistakes, we propose a Dual-Stage Reweighted Mixture-of-Experts (DR-MoE) framework. In the first stage, features are extracted using a frozen ViViT model and a LoRA-tuned ViViT model, which are combined through a feature-level expert module. In the second stage, three classifiers are trained with different objectives: reweighted cross-entropy to mitigate class imbalance, AUC loss to improve ranking under skewed distributions, and label-aware loss with sharpness-aware minimization to enhance calibration and generalization. Their predictions are fused using a classification-level expert module. The proposed method achieves strong performance, particularly in identifying rare and ambiguous mistake instances. The code is available at https://github.com/boyuh/DR-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。