直接处理原始视频帧，实现动作分割端到端训练
End-to-End Action Segmentation Transformer
- 用轻量适配器微调大模型骨干网络
- 通过粗采样视频生成动作候选框提升效率
- 新数据增强策略提升模型泛化能力
当前动作分割方法多依赖其他任务预训练模型提取的帧特征，且仅关注帧级编码与标注，未显式建模动作片段。为克服此局限，本文提出端到端动作分割变换器（EAST），可直接处理原始视频帧，无需预提取特征，支持真正端到端训练。主要贡献包括：(1) 轻量级适配器设计，实现大模型骨干网络高效微调；(2) 基于动作候选框的分割-检测框架，利用粗采样视频预测候选区域；(3) 新型基于动作候选框的数据增强策略。EAST 在 GTEA、50Salads、Breakfast 及 Assembly-101 等标准基准上达到最先进性能。
原文摘要 · Abstract (English)
Most recent work on action segmentation relies on pre-computed frame features from models trained on other tasks and typically focuses on framewise encoding and labeling without explicitly modeling action segments. To overcome these limitations, we introduce the End-to-End Action Segmentation Transformer (EAST), which processes raw video frames directly -- eliminating the need for pre-extracted features and enabling true end-to-end training. Our contributions are as follows: (1) a lightweight adapter design for effective fine-tuning of large backbones; (2) an efficient segmentation-by-detection framework for leveraging action proposals predicted over a coarsely downsampled video; and (3) a novel action-proposal-based data augmentation strategy. EAST achieves SOTA performance on standard benchmarks, including GTEA, 50Salads, Breakfast, and Assembly-101.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。