通过探针分析优化小模型缩放，实现高效低参推理。
ProbeScale: Probing Analysis to Optimize Neural Scaling Laws for Efficient Small Language Model Inference
- 用任务探针量化各层重要性，选最优子网络
- 压缩5到10倍参数，性能保留95%~98%
- 适合资源受限场景下的小模型部署
小语言模型（SLMs）在能力与计算可行性间取得平衡。神经缩放定律为训练提供指导，表明其内部表征随规模增长而增强。然而，即使在严格资源约束下部署这些模型也具挑战性。语言模型探针可用于分析模型内部编码的语言知识。本文提出ProbScale框架，融合缩放定律与探针分析，识别预训练小模型中的参数高效子网络。ProbScale利用高质量的已缩放小模型表示，通过任务特异性探针数学量化每层对目标下游能力的相关性，从而在参数预算下选择性能最优的子网络组合。实验表明，在RoBERTa-Large和T5-Base等代表性模型上，ProbScale可实现5至10倍的参数缩减，同时在目标任务上保持原模型95%至98%的性能，优于启发式基线方法。
原文摘要 · Abstract (English)
Small Language Models (SLMs) offer a balance between capability and computational feasibility. Neural scaling laws inform their optimal training, suggesting that they possess rich internal representations that scale with their size. However, deploying even these SLMs can be challenging under strict resource constraints. Language model probing provides methods for analyzing the linguistic knowledge encoded in a model's internals. We propose ProbScale, a framework that unifies insights from scaling laws and probing to identify parameter-efficient subnetworks within pre-trained SLMs. ProbScale utilizes the high-quality representations of well-scaled SLMs and uses task-specific probes to mathematically quantify the relevance of each layer for target downstream capabilities. This allows selecting subnetworks that optimally trade off performance against parameter size. We formulate the subnetwork selection as finding a layer subset maximizing aggregated, task-weighted probe performance under a parameter budget. Experiments on representative SLMs such as RoBERTa-Large and T5-Base demonstrate that ProbScale identifies subnetworks achieving significant parameter reduction, from 5 to 10 times, while maintaining high performance (95% to 98% of the original SLMs) on targeted tasks, outperforming heuristic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。