优化视觉Transformer训练,提升脑活动预测准确率。
Training Compute-Optimal Vision Transformers for Brain Encoding
- 通过调整模型隐藏层维度和数据规模,显著提升脑编码性能。
- 更大数据集(600万)和更大隐藏层带来更高相关性,效果优于增加层数或注意力头数。
- 16位浮点精度可节省1.17倍训练时间,适合高性能计算场景。
视觉Transformer在脑编码中的最优训练受模型规模、数据规模和计算资源三方面影响。本研究聚焦数据扩展、模型扩展与高性能计算对脑编码结果的影响。采用VideoGPT提取视频的高效时空特征,并用岭回归模型基于这些特征预测脑活动,对比了不同数据规模(1万、10万、100万、600万)及GPT-2多种配置(隐藏层维度、层数、注意力头数)下的表现。还比较了32位与16位浮点精度的训练效果。结果表明,增大隐藏层维度能显著提升脑编码性能,所有被试的皮尔逊相关系数均提高;而注意力头数对结果影响不显著。层数增加有一定改善,但不如隐藏层维度稳定。数据规模越大,编码性能越好,最大数据集(600万)达到最高相关系数。数据扩展的效果比模型扩展更显著。此外,16位精度训练在保持32位精度相同编码准确率的同时,将训练时间缩短1.17倍,适用于高性能计算任务。
原文摘要 · Abstract (English)
The optimal training of a vision transformer for brain encoding depends on three factors: model size, data size, and computational resources. This study investigates these three pillars, focusing on the effects of data scaling, model scaling, and high-performance computing on brain encoding results. Using VideoGPT to extract efficient spatiotemporal features from videos and training a Ridge model to predict brain activity based on these features, we conducted benchmark experiments with varying data sizes (10k, 100k, 1M, 6M) and different model configurations of GPT-2, including hidden layer dimensions, number of layers, and number of attention heads. We also evaluated the effects of training models with 32-bit vs 16-bit floating point representations. Our results demonstrate that increasing the hidden layer dimensions significantly improves brain encoding performance, as evidenced by higher Pearson correlation coefficients across all subjects. In contrast, the number of attention heads does not have a significant effect on the encoding results. Additionally, increasing the number of layers shows some improvement in brain encoding correlations, but the trend is not as consistent as that observed with hidden layer dimensions. The data scaling results show that larger training datasets lead to improved brain encoding performance, with the highest Pearson correlation coefficients observed for the largest dataset size (6M). These findings highlight that the effects of data scaling are more significant compared to model scaling in enhancing brain encoding performance. Furthermore, we explored the impact of floating-point precision by comparing 32-bit and 16-bit representations. Training with 16-bit precision yielded the same brain encoding accuracy as 32-bit, while reducing training time by 1.17 times, demonstrating its efficiency for high-performance computing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。