arXiv:2502.07783cs.LG2025-02NeurIPS被引 3

通过调节激活函数曲率,用单个参数无训练地控制模型决策边界。

Curvature Tuning: Provable Training-free Model Steering From a Single Parameter

  • 用单个超参数修改激活函数,实现可解释的模型调控。
  • 在12个数据集上提升ResNet-50/152准确率8.59%/8.34%(相比线性探测)。
  • 显著增强模型鲁棒性,对抗攻击下准确率提升超1000倍,适合高效微调场景。

模型与数据规模的增长重塑了人工智能格局,微调预训练模型已成为解决下游任务的标准范式。然而,主流微调方法通常依赖权重调整,缺乏可解释性且依赖启发式超参数。本文另辟蹊径,将焦点从权重转向激活函数,将其视为样条算子。提出曲率调优(Curvature Tuning, CT),一种可解释且理论严谨的无训练模型调控方法:通过向激活函数注入单个超参数,调节模型决策边界的曲率。我们证明CT能严格改变决策边界曲率,并从根本上将模型投影到光滑函数空间——这与现有微调方法以特征适应为主形成互补。令该超参数可训练,可得到一种全新、高度参数高效的微调方法。实验表明,CT同时提升泛化性与鲁棒性:在12个数据集上,对ResNet-50/152的下游准确率分别比线性探测提升8.59%/8.34%,比LoRA提升4.64%/1.70%;在RobustBench的ℓ∞基准上,鲁棒准确率提升1032.64%/1494.46%。代码已公开于https://github.com/Leon-Leyang/curvature-tuning。

原文摘要 · Abstract (English)

The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation, often lack interpretability, and depend on heuristically chosen hyperparameters. In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model's decision boundary by injecting a single hyperparameter into its activation functions. We show that CT provably adjusts model decision boundary curvature and, more fundamentally, projects a model onto a space of smooth functions-thereby complementing current finetuning methods, whose effect lies primarily in feature adaptation. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. Empirically, CT improves both generalization and robustness. For example, it boosts downstream accuracy of ResNet-50/152 by 8.59%/8.34% over linear probing and 4.64%/1.70% over LoRA across 12 datasets, and improves robust accuracy on the $\ell_\infty$ benchmark from RobustBench by 1032.64%/1494.46%. Our code is available at https://github.com/Leon-Leyang/curvature-tuning.

模型调控无训练微调曲率控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。