用结构化PCA压缩模型输入，大幅减小参数量且不损失性能。
Compressing Large Language Models with PCA Without Performance Loss
- 对图像或文本序列分段应用结构化PCA，实现高效压缩。
- 压缩后模型在多个数据集上保持高准确率，参数量降至原系统的17%以下。
- 适合追求轻量化部署的开发者，尤其适用于资源受限场景。
我们证明，通过结构化方式应用主成分分析（PCA），无论是对极坐标变换后的图像，还是对标记序列分段处理，都能在不牺牲性能的前提下实现神经模型的极端压缩。在三个案例中，基于极坐标变换后的MNIST训练的一层分类器，仅用840个参数即达到超过98%的准确率；在70维PCA降维后的MiniLM嵌入上训练的两层Transformer，在20 Newsgroups数据集上达到76.62%准确率，参数量仅81,000；一个解码器仅使用70维PCA嵌入，生成连贯的标记序列，与完整MiniLM表示的余弦相似度保持在97%以上，参数量低于GPT-2的17%。这些结果表明，基于PCA的输入压缩是一种通用而高效的策略，可使模型容量与信息内容精准匹配，适用于多种模态的轻量级架构设计。
原文摘要 · Abstract (English)
We demonstrate that Principal Component Analysis (PCA), when applied in a structured manner, either to polar-transformed images or segment-wise to token sequences, enables extreme compression of neural models without sacrificing performance. Across three case studies, we show that a one-layer classifier trained on PCA-compressed polar MNIST achieves over 98 percent accuracy using only 840 parameters. A two-layer transformer trained on 70-dimensional PCA-reduced MiniLM embeddings reaches 76.62 percent accuracy on the 20 Newsgroups dataset with just 81000 parameters. A decoder-only transformer generates coherent token sequences from 70-dimensional PCA embeddings while preserving over 97 percent cosine similarity with full MiniLM representations, using less than 17 percent of the parameter count of GPT-2. These results highlight PCA-based input compression as a general and effective strategy for aligning model capacity with information content, enabling lightweight architectures across multiple modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。