用核范数快速评估大模型的信息压缩能力，速度比传统方法快8到24倍。
Large Language Model Evaluation via Matrix Nuclear-Norm
- 用L1,2范数近似矩阵核范数，避免奇异值分解，降低计算复杂度。
- 在CEREBRAS-GPT上测试，模型越大提速越明显，6.7B参数时快24倍。
- 适合需要高效评估大模型压缩与多样性能力的研究者和工程团队。
随着大语言模型（LLMs）不断发展，高效的评估指标对衡量其信息压缩与冗余减少能力至关重要。传统指标如矩阵熵虽有洞察力，但因奇异值分解（SVD）带来的O(n³)时间复杂度，在大规模模型上计算成本过高。为此，本文提出矩阵核范数，不仅可量化模型的信息压缩能力，还通过凸近似矩阵秩，同时捕捉预测区分度与多样性。进一步采用L1,2-范数近似核范数，将时间复杂度降至O(n²)，无需SVD计算。实验显示，该方法在CEREBRAS-GPT模型上，当参数量从111M增至6.7B时，速度比矩阵熵提升8至24倍；在Pythia等其他模型上也验证了其高效性。基准测试与模型输出评估表明，矩阵核范数是一种可靠、可扩展且高效的评估工具，兼顾准确性与效率。代码已开源：https://github.com/MLGroupJLU/MatrixNuclearNorm。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to evolve, efficient evaluation metrics are vital for assessing their ability to compress information and reduce redundancy. While traditional metrics like Matrix Entropy offer valuable insights, they are computationally intensive for large-scale models due to their \( O(n^3) \) time complexity with Singular Value Decomposition (SVD). To mitigate this issue, we introduce the Matrix Nuclear-Norm, which not only serves as a metric to quantify the data compression proficiency of LLM but also provides a convex approximation of matrix rank to capture both predictive discriminability and diversity. By employing the \( L_{1,2}\text{-norm} \) to further approximate the nuclear norm, we can effectively assess the model's information compression capabilities. This approach reduces the time complexity to \( O(n^2) \) and eliminates the need for SVD computation. Consequently, the Matrix Nuclear-Norm achieves speeds 8 to 24 times faster than Matrix Entropy for the CEREBRAS-GPT model as sizes increase from 111M to 6.7B. This performance gap becomes more pronounced with larger models, as validated in tests with other models like Pythia. Additionally, evaluations on benchmarks and model responses confirm that our proposed Matrix Nuclear-Norm is a reliable, scalable, and efficient tool for assessing LLMs' performance, striking a balance between accuracy and computational efficiency. The code is available at https://github.com/MLGroupJLU/MatrixNuclearNorm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。