用特征分布对齐合并多个评分模型,节省一半存储且更准确。
Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment
- 基于格罗莫夫-沃瑟斯坦距离度量特征分布相似性,选择最匹配的模型对进行合并。
- 合并后模型在微F1、精确匹配率等指标上均优于基线方法,显著提升评分准确性。
- 适合需要多任务高效部署的教育AI系统,尤其关注存储与性能平衡的场景。
自动评分能提升教育效率,但为每项任务部署独立神经网络会增加存储、维护成本和冗余计算。本文提出基于格罗莫夫-沃瑟斯坦特征对齐的模型合并方法(GW-SMM),通过提取各任务模型对学生作答的特征表示,利用格罗莫夫-沃瑟斯坦距离衡量特征分布相似性,识别最适配的模型对或三元组进行合并。仅合并分类头之前的共享层,保留各任务独立的分类头,形成统一特征提取器。实验对比人工评分与GPT-o1-based合并方法,GW-SMM在微F1、宏F1、精确匹配率及标签级准确率上均表现更优,其中微F1与标签准确率相比GPT-o1方法显著提升(p=0.04, p=0.01)。同时,模型体积减半,存储开销降低50%,兼顾高效与高精度。
原文摘要 · Abstract (English)
Automatic scoring of student responses enhances efficiency in education, but deploying a separate neural network for each task increases storage demands, maintenance efforts, and redundant computations. To address these challenges, this paper introduces the Gromov-Wasserstein Scoring Model Merging (GW-SMM) method, which merges models based on feature distribution similarities measured via the Gromov-Wasserstein distance. Our approach begins by extracting features from student responses using individual models, capturing both item-specific context and unique learned representations. The Gromov-Wasserstein distance then quantifies the similarity between these feature distributions, identifying the most compatible models for merging. Models exhibiting the smallest pairwise distances, typically in pairs or trios, are merged by combining only the shared layers preceding the classification head. This strategy results in a unified feature extractor while preserving separate classification heads for item-specific scoring. We validated our approach against human expert knowledge and a GPT-o1-based merging method. GW-SMM consistently outperformed both, achieving a higher micro F1 score, macro F1 score, exact match accuracy, and per-label accuracy. The improvements in micro F1 and per-label accuracy were statistically significant compared to GPT-o1-based merging (p=0.04, p=0.01). Additionally, GW-SMM reduced storage requirements by half without compromising much accuracy, demonstrating its computational efficiency alongside reliable scoring performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。