用一个微型模型一次性优化多个大模型，降低调优成本。
Cross-model Control: Improving Multiple Large Language Models in One-time Training
- 用小型模型学习并修改大模型的输出逻辑。
- 在指令微调和去敏感信息任务中提升效果，仅一次训练即可应用。
- 适合希望低成本批量优化大模型的研究者或工程师。
大型语言模型（LLMs）的数量持续增长，其参数规模和词表各不相同。尽管性能强大，但它们常需针对特定需求进行优化，如遵循指令或避免输出真实世界敏感信息。然而，如何复用一个模型的微调成果以减少其他模型的训练成本仍是一个挑战。为此，我们提出跨模型控制（Cross-model Control, CMC），通过一个可移植的小型语言模型，在一次训练中同时改进多个大模型。我们观察到不同模型在微调前后的对数偏移具有显著相似性。基于此，引入一个参数极少的小型模型，与冻结的模板大模型共同训练，使其学会调整大模型的输出对数。为使该小型模型适用于不同词表的模型，我们设计了一种名为PM-MinED的新令牌映射策略。我们在指令微调和去敏感信息任务上进行了大量实验，验证了CMC的有效性。代码已公开于https://github.com/wujwyi/CMC。
原文摘要 · Abstract (English)
The number of large language models (LLMs) with varying parameter scales and vocabularies is increasing. While they deliver powerful performance, they also face a set of common optimization needs to meet specific requirements or standards, such as instruction following or avoiding the output of sensitive information from the real world. However, how to reuse the fine-tuning outcomes of one model to other models to reduce training costs remains a challenge. To bridge this gap, we introduce Cross-model Control (CMC), a method that improves multiple LLMs in one-time training with a portable tiny language model. Specifically, we have observed that the logit shift before and after fine-tuning is remarkably similar across different models. Based on this insight, we incorporate a tiny language model with a minimal number of parameters. By training alongside a frozen template LLM, the tiny model gains the capability to alter the logits output by the LLMs. To make this tiny language model applicable to models with different vocabularies, we propose a novel token mapping strategy named PM-MinED. We have conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC. Our code is available at https://github.com/wujwyi/CMC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。