首次验证运动生成模型的缩放定律，可预测最优模型规模
ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model
- 构建文本前缀自回归运动生成框架，含Motion FSQ-VAE分词器
- 发现测试损失随算力呈对数关系，参数与数据量符合作品幂律
- 基于定律预测1e18算力下的最优配置，实测结果完全吻合
缩放定律已在自然语言处理和计算机视觉等领域得到验证，但在动作生成领域仍待探索。本文提出一个可扩展的动作生成框架，包含运动分词器Motion FSQ-VAE和文本前缀自回归变压器。通过全面实验，我们观察到该系统的缩放行为。首次证实动作生成中存在缩放定律：前缀自回归模型的归一化测试损失与算力预算呈对数关系；非词汇参数、词汇参数和数据标记数分别与算力预算符合幂律关系。利用该定律，我们预测了1e18算力预算下的最优变压器规模、词汇大小及数据需求。当模型以最优配置训练时，测试损失与预测值精确一致，验证了缩放定律的有效性。
原文摘要 · Abstract (English)
The scaling law has been validated in various domains, such as natural language processing (NLP) and massive computer vision tasks; however, its application to motion generation remains largely unexplored. In this paper, we introduce a scalable motion generation framework that includes the motion tokenizer Motion FSQ-VAE and a text-prefix autoregressive transformer. Through comprehensive experiments, we observe the scaling behavior of this system. For the first time, we confirm the existence of scaling laws within the context of motion generation. Specifically, our results demonstrate that the normalized test loss of our prefix autoregressive models adheres to a logarithmic law in relation to compute budgets. Furthermore, we also confirm the power law between Non-Vocabulary Parameters, Vocabulary Parameters, and Data Tokens with respect to compute budgets respectively. Leveraging the scaling law, we predict the optimal transformer size, vocabulary size, and data requirements for a compute budget of $1e18$. The test loss of the system, when trained with the optimal model size, vocabulary size, and required data, aligns precisely with the predicted test loss, thereby validating the scaling law.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。