arXiv:2411.04967cs.CVcs.LG2024-11NeurIPS被引 7

提出一种前后不对称的混合网络,兼顾速度与性能。

AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation

  • 前段用卷积后段用Transformer,结构设计更高效。
  • 在图像识别生成任务中表现优于现有模型,推理更快。
  • 适合需要快速部署的多任务视觉应用。

神经网络架构设计需权衡多项关键因素:相似决策可复用于多种任务,具备良好的延迟与性能平衡,支持多样任务、可扩展至大规模数据与算力,且适配不同硬件。为此,我们提出AsCAN——一种融合卷积与Transformer块的混合架构。通过重新审视混合架构设计原则,提出一种简单有效的非对称结构:早期阶段以更多卷积块为主,后期逐渐过渡为更多Transformer块。AsCAN支持图像识别、分割、类别条件图像生成等多样化任务,并在性能与延迟之间取得优异权衡。进一步将同一架构扩展至大规模文本到图像生成任务,其性能超越近期公开及商业模型。值得注意的是,即使未对Transformer块进行计算优化,其推理速度仍快于采用高效注意力机制的现有工作,凸显该方法的优势与价值。

原文摘要 · Abstract (English)

Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN -- a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach.

混合架构图像生成高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。