手把手重实现GPT核心结构，从零开始训练字符级文本生成。
MiniGPT: Rebuilding GPT from First Principles
- 从零构建GPT核心模块，包含自注意力与前馈网络的完整实现。
- 0.83M参数模型在3000轮后验证损失1.7236，10.77M模型达1.4780并生成莎士比亚风格对话。
- 适合想理解GPT底层原理、动手实践的开发者或学生。
本文提出MiniGPT，一个基于PyTorch的轻量级GPT式自回归语言建模实现。在学习Andrej Karpathy的nanoGPT设计后，从零重建GPT核心流程，所有代码独立编写于单一笔记本中。MiniGPT实现字符级分词、位置嵌入、因果多头自注意力、预层归一化变换块、残差连接、前馈MLP、下一轮词交叉熵训练（教师强制）、验证跟踪、检查点选择及自回归文本生成。在Tiny Shakespeare数据集上评估：0.83M参数基线模型经3000轮训练后验证损失为1.7236；更强的10.77M参数配置使用更大上下文长度和优化训练设置，最佳验证损失降至1.4780，并能生成具有明显莎士比亚风格对话结构的文本。MiniGPT不引入新架构，而是清晰记录从原始文本到训练完成的可复现实现路径，涵盖设计选择、训练行为、生成质量与实际限制。
原文摘要 · Abstract (English)
This paper presents MiniGPT, a compact from-scratch implementation of GPT-style autoregressive language modeling in PyTorch. The aim is to rebuild the core GPT pipeline from first principles after studying the design of nanoGPT by Andrej Karpathy, while keeping the model and training code independently written in a single notebook. MiniGPT implements token and positional embeddings, causal multi-head self-attention, pre-LayerNorm Transformer blocks, residual connections, feed-forward MLP layers, next-token cross-entropy training (teacher forcing), validation tracking, checkpoint selection, and autoregressive text generation. This paper evaluates the implementation on Tiny Shakespeare dataset using character-level tokenization. A baseline 0.83M-parameter model reaches a validation loss of 1.7236 after 3000 training iterations. A stronger 10.77M-parameter configuration, using a larger context length and improved training settings, reaches a best validation loss of 1.4780 and generates text with recognizable Shakespeare-style dialogue structure. MiniGPT does not introduce a new language-model architecture. Instead, it documents a clear and reproducible implementation path from raw text to trained character-level generation, including design choices, training behavior, generation quality, and practical limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。