用GPT-2预处理结构化文本，让Gzip压缩效率提升5.8%。
GPT-2 as a Compression Preprocessor: Improving Gzip for Structured Text Domains
- 用GPT-2重写结构化文本，增强语法重复性以适配Gzip
- 在防御日志上压缩率提升0.34%，HTML文件提升5.8%
- 适合医疗、日志、网页等结构化数据压缩场景
当前大量领域数据（如医疗记录、防御日志、HTML网页）持续生成，需高效压缩。传统压缩算法如Gzip依赖二进制模式重复性，但结构化格式（如JSON、XML、HTML）虽具语义重复，缺乏语法重复，导致压缩效果差。本文提出基于GPT-2的预处理器，将此类文件转换为更易被Gzip压缩的格式。实验使用真实与合成的日志、HTML数据，结果显示防御日志压缩率提升0.34%，HTML文件提升5.8%，验证了该方法在结构化文本域的有效性。
原文摘要 · Abstract (English)
In the modern era, large volumes of data are being produced continuously, especially in domain-specific fields such as medical records and clinical files, defence logs and HTML-based web traffic. Data with such volume and complexity needs to be compressed before storing and transmitting efficiently. Data compression has gained significant attention from modern researchers, resulting in the development of fast and efficient compression algorithms such as Gzip. However, since gzip works on the principle of repetition of binary patterns, one of the limitations of gzip is that domain-specific formats like JSON, XML, HTML, and log files, while structured, may have semantic repetition but not syntactic repetition, which gzip finds difficult to compress. In this article, we propose a GPT-based preprocessor for such domain-specific files. We propose a pipeline made up of GPT-2 taking domain-specific files as input, which pattern-based compressors like gzip find difficult to work on. The preprocessor results are output in a file that is designed for compressors like gzip. After preprocessing, the gzip works on the other end of the pipeline and compresses the data as usual. We used different types of both real-world and synthetically generated data, such as logs and HTML files, for the experiment of the proposed model. We found promising results and an improvement of the Defence logs by 0.34 per cent and HTML files by 5.8 per cent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。