从大模型中提取任务专用小模型，提速且更易理解
Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference
- 通过可解释性技术定位完成特定任务的神经电路
- 模型参数减少最高达82.77%，无需重新训练
- 适合资源受限场景下的高效推理应用
大型语言模型（LLM）在多种任务上表现优异，但其规模持续增长，限制了在计算资源受限环境中的应用。尽管具备通用能力，但在仅执行单一任务时，其余功能成为冗余负担。本文提出一种新方法，自动提取能够独立完成特定任务的最小模型子集，无需额外训练且仅需少量数据样本。实验表明，所提取模型参数量最多减少82.77%，同时聚焦于特定任务的神经电路，显著提升可解释性，便于使用机制可解释性（MI）技术分析。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown impressive performance across a wide range of tasks. However, the size of LLMs is steadily increasing, hindering their application on computationally constrained environments. On the other hand, despite their general capabilities, there are many situations where only one specific task is performed, rendering all other capabilities unnecessary and wasteful. This leads us to the following question: Is it possible to extract the minimal subset from an LLM that is able to perform a specific task in a faster, standalone manner? Recent works on Mechanistic Interpretability (MI) have shown that specific tasks are performed by a localized subset of components, or circuit. However, current techniques used to identify the circuit cannot be used to extract it for its standalone usage. In this work, we propose a novel approach to automatically extract the subset of the LLM that properly performs a targeted task requiring no additional training and a small amount of data samples. We evaluate our approach on different tasks and show that the resulting models are (i) considerably smaller, reducing the number of parameters up to 82.77% and (ii) more interpretable, as they focus on the circuit that is used to carry out the specific task, and can therefore be understood using MI techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。