构建模块化故障注入框架,评估大模型在硬件故障下的鲁棒性。
GoldenTransformer: A Modular Fault Injection Framework for Transformer Robustness Research
- 基于PyTorch和HuggingFace,支持权重、激活、注意力等多类故障注入
- 可在分类与生成任务中验证模型对故障的敏感度,支持可复现实验
- 适合研究大模型可靠性及实际部署中系统设计的开发者
Transformer已成为自然语言处理、计算机视觉等多个领域先进模型的基础。尽管广泛应用,其在故障条件下的鲁棒性仍缺乏深入研究。本文提出GoldenTransformer,一个模块化且可扩展的故障注入框架,用于评估大型语言模型在人为引入硬件故障下的韧性。该框架提供统一的Python平台,支持在预训练Transformer模型中注入多种故障类型,包括权重破坏、激活干扰和注意力层扰动。受DNN黄金眼模拟器启发,框架特别考虑了大型Transformer架构的结构复杂性、潜在依赖关系及非均匀层定义等挑战。GoldenTransformer基于PyTorch与HuggingFace Transformers构建,原生支持实验可复现性、指标记录与可视化。我们详细阐述其技术设计,并通过多个分类与生成任务的示例实验展示其应用。通过在Transformer的多个逻辑与结构层级上实现可控故障注入,GoldenTransformer为研究人员和实践者提供了分析模型鲁棒性的有力工具,有助于指导真实大模型应用中的可靠系统设计。
原文摘要 · Abstract (English)
Transformers have become the foundation for a wide range of state--of--the--art models across natural language processing, computer vision, and other machine learning domains. Despite their widespread deployment, the robustness of these models under fault conditions remains underexplored. We present GoldenTransformer, a modular and extensible fault injection framework designed to evaluate the resiliency of Large Language Models to induced hardware faults. GoldenTransformer offers a unified Python-based platform for injecting diverse classes of faults--such as weight corruption, activation injections, and attention--level disruptions--into pretrained transformer--based models. Inspired by the GoldenEye simulator for DNNs, our framework focuses on the unique challenges of working with large transformer architectures, including considerations such as structural complexity, latent dependencies, and nonuniform layer definitions. GoldenTransformer is built atop PyTorch and HuggingFace Transformers, and it supports experiment reproducibility, metric logging, and visualization out of the box. We detail the technical design and use of GoldenTransformer and demonstrate through several example experiments on classification and generation tasks. By enabling controlled injection of faults at multiple logical and structural points in a transformer, GoldenTransformer offers researchers and practitioners a valuable tool for model robustness analysis and for guiding dependable system design in real-world LLM applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。