模型融合可实现可信泛化，小样本下仍能保证有效性能。
Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning
- 通过模型融合替代微调，降低泛化误差
- 100个样本即可获得非平凡的泛化保证
- 适用于视觉与语言大模型，适合高风险场景
在医疗、安全等高风险领域，深度网络的独立同分布泛化能力认证是信任AI的关键前提。然而，现有泛化界在小样本情况下常为空洞（vacuous）。本文首次揭示模型融合方法与泛化证书之间的新联系，表明仅需微调现有策略，即可获得非空泛化保证。关键在于以数据驱动的融合方式学习下游任务，而非传统微调，使泛化差距极小且与基础网络规模无关。实验表明，使用VIT-B和mistral-7B等模型，在仅100个样本下即可获得非平凡泛化保证，这对现有系统的可信认证具有直接意义，并为理论与实践结合开辟新方向。
原文摘要 · Abstract (English)
Certifying the IID generalisation ability of deep networks is the first of many requirements for trusting AI in high-stakes applications from medicine to security. However, when instantiating generalisation bounds for deep networks it remains challenging to obtain non-vacuous guarantees, especially when applying contemporary large models on the small scale data prevalent in such high-stakes fields. In this paper, we draw a novel connection between a family of learning methods based on model fusion and generalisation certificates, and surprisingly show that with minor adjustment several existing learning strategies already provide non-trivial generalisation guarantees. Essentially, by focusing on data-driven learning of downstream tasks by fusion rather than fine-tuning, the certified generalisation gap becomes tiny and independent of the base network size, facilitating its certification. Our results show for the first time non-trivial generalisation guarantees for learning with as low as 100 examples, while using vision models such as VIT-B and language models such as mistral-7B. This observation is significant as it has immediate implications for facilitating the certification of existing systems as trustworthy, and opens up new directions for research at the intersection of practice and theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。