为视频动作识别设计异构双流架构，让不同模态用合适模型并智能融合。
A Heterogeneous Two-Stream Framework for Video Action Recognition with Comparative Fusion Analysis
- RGB和光流分别用ViT-Tiny和MobileNetV2，适配各自数据特性。
- 在UCF11上达98.12%准确率，跨模态注意力效果显著；UCF50上加权融合最稳。
- 发现数据集越大越依赖外观信息，融合策略需随规模调整。
多数双流动作识别网络对RGB与光流流使用相同卷积主干，忽视二者本质差异：光流捕捉细微运动模式，而RGB帧包含丰富外观与场景上下文。本文提出DualStreamHybrid，一种异构双流架构，为每一流分配适配其输入的主干：对RGB帧使用预训练ViT-Tiny/16，对20通道堆叠光流表示则使用从头训练的MobileNetV2。通过学习的投影层将两个不同尺寸的特征向量映射到统一维度后融合，实现两流交互而不强制结构对称。在统一框架内设计五种融合策略——晚期融合、拼接、交叉注意力、加权融合、门控融合，并在UCF11（1,600视频，11类）和UCF50（6,681视频，50类）上评估其在不同数据规模下的表现。在UCF11上，交叉注意力达98.12%测试准确率，优于仅用RGB的ViT-Tiny基线（95.94%），表明显式跨模态注意力在小而简单的数据集上更有效。在UCF50上，加权融合达到96.86%，表现最一致。学习到的流权重显示：UCF11中两模态贡献近乎相等（RGB: 0.507，光流: 0.493），而UCF50略微偏好RGB流（RGB: 0.554，光流: 0.446），反映更大、更视觉多样的动作空间。整体表明，即使轻量级运动流也能有效补充强外观编码器，且最优融合策略取决于数据集规模。
原文摘要 · Abstract (English)
Most two-stream action recognition networks apply the same convolutional backbone to both RGB and optical flow streams, ignoring the fact that the two modalities have fundamentally different structural properties. Optical flow captures fine-grained motion patterns, while RGB frames carry rich appearance and scene context - treating them identically discards this distinction. We propose DualStreamHybrid, a heterogeneous two-stream architecture that assigns each stream a backbone suited to its input: a pretrained ViT-Tiny/16 for RGB frames, and a MobileNetV2 trained from scratch on a 20-channel stacked optical flow representation. A learned projection layer maps the two differently-sized feature vectors to a common dimensionality before fusion, enabling the two streams to interact without forcing architectural symmetry. We design five fusion strategies within a unified framework - late fusion, concatenation, cross-attention, weighted fusion, and gated fusion - and evaluate them on UCF11 (1,600 videos, 11 classes) and UCF50 (6,681 videos, 50 classes) to study how fusion behaviour scales with dataset size. On UCF11, cross-attention achieves 98.12% test accuracy, outperforming the RGB-only ViT-Tiny baseline of 95.94%, which suggests that explicit inter-modal attention is particularly effective on smaller, less complex datasets. On UCF50, weighted fusion reaches 96.86% and proves the most consistent strategy across both benchmarks. The learned stream weights reveal an interesting pattern: UCF11 sees near-equal modality contribution (RGB: 0.507, flow: 0.493), while UCF50 favours the RGB stream slightly more (RGB: 0.554, flow: 0.446) - arguably reflecting the larger and more visually diverse action space. Taken together, these results suggest that even a lightweight motion stream meaningfully complements a strong appearance encoder, and that the optimal fusion strategy depends on dataset scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。