通过分析Hugging Face模型演进数据,建立评估机器学习系统语义保持的实证框架。
An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face
- 从Hugging Face提取模型版本历史、卡片和性能指标,构建演化数据集。
- 发现性能变化可揭示语义漂移,识别出常见重构模式。
- 适合关注ML系统可维护性与可信性的研究人员与工程师。
随着机器学习(ML)成为高自主系统的核心,保障学习型软件系统(LESS)的可信性至关重要。然而,ML的非确定性和运行时定义的语义使传统软件重构变得复杂。本文将LESS中的语义保持定义为:智能组件优化不改变系统的整体功能行为。提出一个实证框架,通过挖掘Hugging Face的模型演化数据来评估语义保持。从大量模型中提取提交历史、模型卡片和性能指标,以三个领域为案例,追踪版本间性能变化。分析表明,可通过提交间的评估指标检测语义漂移,并基于提交信息识别常见重构模式。尽管受API限制未能建立全规模阈值,但该流程为定义社区认可的语义保持边界提供了基础。贡献包括:(1)基于170万条Hugging Face条目,通过可复现管道使用原生HF API构建的大规模模型演化数据集;(2)针对536个模型和4000+指标的实用评估流程;(3)展示语义漂移实际表现的实证案例。这些成果推动了更可维护、更可信的ML系统发展。
原文摘要 · Abstract (English)
As machine learning (ML) becomes an integral part of high-autonomy systems, it is critical to ensure the trustworthiness of learning-enabled software systems (LESS). Yet, the nondeterministic and run-time-defined semantics of ML complicate traditional software refactoring. We define semantic preservation in LESS as the property that optimizations of intelligent components do not alter the system's overall functional behavior. This paper introduces an empirical framework to evaluate semantic preservation in LESS by mining model evolution data from HuggingFace. We extract commit histories, $\textit{Model Cards}$, and performance metrics from a large number of models. To establish baselines, we conducted case studies in three domains, tracing performance changes across versions. Our analysis demonstrates how $\textit{semantic drift}$ can be detected via evaluation metrics across commits and reveals common refactoring patterns based on commit message analysis. Although API constraints limited the possibility of estimating a full-scale threshold, our pipeline offers a foundation for defining community-accepted boundaries for semantic preservation. Our contributions include: (1) a large-scale dataset of ML model evolution, curated from 1.7 million Hugging Face entries via a reproducible pipeline using the native HF hub API, (2) a practical pipeline for the evaluation of semantic preservation for a subset of 536 models and 4000+ metrics and (3) empirical case studies illustrating semantic drift in practice. Together, these contributions advance the foundations for more maintainable and trustworthy ML systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。