从大模型一键生成小而专的Transformer，性能更优且适合设备端部署。
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
- 通过参数映射将大模型知识迁移至小模型，专用于特定任务。
- 生成的小模型在图像任务上表现优于通用条件模型。
- 适合资源受限场景，如手机、嵌入式设备等边缘计算应用。
现代基础模型通常在涵盖多种数据模态、主题和下游任务的语料上训练，使用这些模型计算开销巨大，多数消费级设备难以承受。此外，大部分通用知识对特定任务而言可能无关紧要。本文探索一种将大型Transformer模型参数映射到小型专用模型参数的技术。通过使转换过程针对特定任务优化，旨在让小型模型仅保留执行特定任务所需的精炼知识。我们在图像建模任务上验证该方法，结果表明生成的小模型性能超越通用条件模型。
原文摘要 · Abstract (English)
Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。