SWAN动态调节多模态模型资源分配，应对运行时变化。
SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations
- 根据输入质量与复杂度动态分配计算资源
- 在预算内降低49%计算量，精度损失极小
- 适合自动驾驶等实时多模态任务
部署在真实环境中的多模态深度神经网络需应对运行时变化：模态质量波动、输入整体复杂度变化及可用平台资源变动。现有模型难以兼顾：自适应网络无法严格遵守计算预算，控制器型网络忽略输入复杂度，静态分配模型则全面失效。为此，我们提出SWAN（Sample and World-Aware Multimodal Network），首个同时实现三重目标的自适应多模态网络。SWAN采用质量感知控制器，按用户指定的最大预算在各模态间分配资源；在此预算内，自适应门控模块依据样本复杂度动态调整层利用率；为进一步优化，还引入令牌丢弃模块，在检测前屏蔽语义无关的多模态特征。我们在自动驾驶场景下评估了该方法，针对复杂的多目标3D检测任务，最大可减少49%的浮点运算量（FLOPs），且精度下降微乎其微。
原文摘要 · Abstract (English)
Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platform resources. Current networks struggle with such fluctuations -- adaptive networks cannot adhere to a strict compute budget, controller-based networks neglect to consider input complexity, and statically provisioned networks fail at all the above. Consequently, they do not extract maximum utility from the expended computational resources. We present SWAN (Sample and World-Aware Multimodal Network), the first adaptive multimodal network that accomplishes all three goals. SWAN employs a quality-aware controller to assign resources among modalities according to a variable user-specified maximum budget. Within this budget, an adaptive gating module further optimizes efficiency by scaling layer utilization according to sample complexity. For further gains, SWAN also employs a token dropping module that masks semantically irrelevant multimodal features before performing detections. We evaluate SWAN in the domain of autonomous driving with complex multi-object 3D detection, reducing FLOPs by up to 49% with minimal degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。