用激活值差异表示微调模型，轻松识别任务与领域。
Delta Activations: A Representation for Finetuned Large Language Models
- 通过对比基础模型与微调模型的内部激活变化，生成向量表示。
- 能准确聚类不同任务和领域的模型，且对数据混合具加性特性。
- 支持少量样本快速建模任务，适合模型选择与合并场景。
大型开源语言模型的成功推动了大量针对特定任务和领域的微调模型的产生。然而，由于元数据不一致和存储结构混乱，这些模型的管理和理解仍具挑战。本文提出Delta Activations，通过测量微调模型相对于基础模型内部激活的变化，将其表示为向量嵌入。该方法能有效按任务和领域聚类，揭示模型分布的内在结构。实验表明，该表示在不同微调设置下具有鲁棒性，并在多个数据集混合时表现出加性特性。此外，我们展示了通过少样本微调即可将任务编码为Delta Activations，并进一步探索其在模型选择与合并中的应用。代码已公开于https://github.com/OscarXZQ/delta_activations，旨在促进公开模型的再利用。
原文摘要 · Abstract (English)
The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We introduce Delta Activations, a method to represent finetuned models as vector embeddings by measuring shifts in their internal activations relative to a base model. This representation allows for effective clustering by domain and task, revealing structure in the model landscape. Delta Activations also demonstrate desirable properties: it is robust across finetuning settings and exhibits an additive property when finetuning datasets are mixed. In addition, we show that Delta Activations can embed tasks via few-shot finetuning, and further explore its use for model selection and merging. We hope Delta Activations can facilitate the practice of reusing publicly available models. Code is available at https://github.com/OscarXZQ/delta_activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。