提出新框架，高效预测大模型微调表现，提升选型准确率。
LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
- 基于神经正切核构建可解释的缩放模型，捕捉微调动态
- 在3个大规模基准上达91.1%选型准确率，计算成本降低88.5%
- 适合需要快速筛选大模型的工业应用与研究场景
开源大模型与下游任务的激增使得高效模型选择成为关键挑战，因无法在计算资源限制下对所有候选模型进行微调。尽管已有进展，但大模型在微调过程中的动态行为如何影响泛化性能仍缺乏系统研究。本文提出一种新型理论框架——LENSLLM，基于神经正切核（NTK）构建修正缩放模型，首次揭示了微调过程中的动态特性，并建立一个帕累托-贝叶斯泛化界。该模型可在多种下游任务中实现高精度性能预测。在3个大规模基准上的实验表明，LENSLLM能达到最高91.1%的选型准确率，同时将计算成本降低高达88.5%，优于5种现有先进方法。代码与结果已开源至LensLLM.io。
原文摘要 · Abstract (English)
The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a PAC-Bayesian Generalization Bound that unveils fine-tuning dynamics of LLMs and then introduce LENSLLM, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。