arXiv:2602.00446cs.LGcs.CR2026-02被引 2

让大模型无法被随意微调,同时保持原始性能

Towards Building Non-Fine-Tunable Foundation Models

  • 用稀疏子网络+私有掩码预训练,限制微调空间
  • 实测在多种任务上显著降低未经授权微调的效果
  • 适合需保护模型安全与经济利益的机构使用

开源基础模型虽促进广泛使用,但也使训练者面临经济与安全风险。本文提出非微调型基础模型:模型在发布后仍具通用性,但面对无授权的任务无关微调时仅能获得有限适应收益。我们提出私有掩码预训练(PMP)框架,将表征学习集中在训练早期确定的稀疏子网络中,该二值掩码保密,仅释放最终密集权重。未获掩码的微调将更新与预训练子空间错位的参数,造成微调目标与预训练几何结构的内在不匹配。理论分析表明此不匹配会削弱基于梯度的适应能力,并限制微调收益。大规模语言模型实验验证了PMP在保持基础性能的同时,显著降低各类下游任务中的未经授权微调效果,且非微调能力可通过掩码比例控制。

原文摘要 · Abstract (English)

Open-sourcing foundation models (FMs) enables broad reuse but also exposes model trainers to economic and safety risks from unrestricted downstream fine-tuning. We address this problem by building non-fine-tunable foundation models: models that remain broadly usable in their released form while yielding limited adaptation gains under task-agnostic unauthorized fine-tuning. We propose Private Mask Pre-Training (PMP), a pre-training framework that concentrates representation learning into a sparse subnetwork identified early in training. The binary mask defining this subnetwork is kept private, and only the final dense weights are released. This forces unauthorized fine-tuning without access to the mask to update parameters misaligned with pretraining subspace, inducing an intrinsic mismatch between the fine-tuning objective and the pre-training geometry. We provide theoretical analysis showing that this mismatch destabilizes gradient-based adaptation and bounds fine-tuning gains. Empirical results on large language models demonstrating that PMP preserves base model performance while consistently degrading unauthorized fine-tuning across a wide range of downstream tasks, with the strength of non-fine-tunability controlled by the mask ratio.

基础模型安全机制微调防护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。