arXiv:2505.13425cs.LGcs.AI2025-05被引 9

用专用小模型构建可复用的智能系统,提升垂直领域表现。

Learnware of Language Models: Specialized Small Language Models Can Do Big

  • 通过模型规格描述实现小模型精准匹配与复用
  • 在金融和医疗任务中超越大模型14%以上性能
  • 支持隐私保护协作,适合垂直领域应用

学习软件范式为机器学习提供新思路,使用户无需从头训练即可复用已训练好的模型。该范式依赖模型能力的规格化表示,来识别并利用最适合新任务的模型。尽管该范式已在多个场景验证有效,但在语言模型领域的应用仍不充分。大语言模型虽具备通用问答能力,但在专业领域面临数据稀缺、隐私顾虑和高计算成本问题,因此越来越多专用小语言模型(SLMs)被训练用于特定领域。本文首次系统探索将学习软件范式应用于语言模型,构建了约100个80亿参数的专用领域(金融、医疗、数学)小模型组成的模拟系统,每个模型均配备规格描述,使用户可在不暴露自身数据的前提下,精准匹配最适模型。实验表明,针对每项任务选择一个合适的学习软件进行推理,系统在所有基准测试中均优于基础小模型;在金融任务中,性能超过Qwen1.5-110B、Qwen2.5-72B和Llama3.1-70B-Instruct至少14%;在医疗任务中,优于排名第7的Flan-PaLM-540B。

原文摘要 · Abstract (English)

The learnware paradigm offers a novel approach to machine learning by enabling users to reuse a set of well-trained models for tasks beyond the models' original purposes. It eliminates the need to build models from scratch, instead relying on specifications (representations of a model's capabilities) to identify and leverage the most suitable models for new tasks. While learnware has proven effective in many scenarios, its application to language models has remained largely unexplored. At the same time, large language models (LLMs) have demonstrated remarkable universal question-answering abilities, yet they face challenges in specialized scenarios due to data scarcity, privacy concerns, and high computational costs, thus more and more specialized small language models (SLMs) are being trained for specific domains. To address these limitations systematically, the learnware paradigm provides a promising solution by enabling maximum utilization of specialized SLMs, and allowing users to identify and reuse them in a collaborative and privacy-preserving manner. This paper presents a preliminary attempt to apply the learnware paradigm to language models. We simulated a learnware system comprising approximately 100 learnwares of specialized SLMs with 8B parameters, fine-tuned across finance, healthcare, and mathematics domains. Each learnware contains an SLM and a specification, which enables users to identify the most relevant models without exposing their own data. Experimental results demonstrate promising performance: by selecting one suitable learnware for each task-specific inference, the system outperforms the base SLMs on all benchmarks. Compared to LLMs, the system outperforms Qwen1.5-110B, Qwen2.5-72B, and Llama3.1-70B-Instruct by at least 14% in finance domain tasks, and surpasses Flan-PaLM-540B (ranked 7th on the Open Medical LLM Leaderboard) in medical domain tasks.

小模型学习软件领域适应隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。