用小模型适配器实现万亿参数大模型的个人化定制。
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
- 将适配器作为持久化的个人状态，嵌入共享大模型之上。
- 小适配器在强基础模型下仍能可靠运行，支持百万级个性化模型。
- 适合需要长期记忆与个性化行为的智能应用开发者。
参数高效微调（PEFT）通常被视为全量微调的低成本替代方案。本文提出更广泛的视角：将小型可训练适配器视为强共用基础模型上的持久本地状态。在此框架中，基础模型提供通用能力，而适配器则承载特定实例的行为特征，如偏好、技能、工具习惯和类记忆的更新。研究围绕三个扩展维度展开：规模提升（更强的共享先验使微小本地更新更有效）、规模下降（探索适配器最小化但保持可靠性）、规模扩展（多个持久适配实例共存）。MinT 提供了一个管理适配器身份、版本、来源、评估与服务驻留的基础设施示例。结果表明，PEFT 可作为持久个人模型的紧凑基础，而不仅是全量微调的经济替代方案。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。