用LLaMA3预测文本，只存预测错的词，实现无损压缩并检测数据来源
Llamazip: Leveraging LLaMA for Lossless Text Compression and Training Dataset Detection
- 基于LLaMA3预测能力，仅存储模型无法预测的词元
- 可实现高效无损压缩，且能判断文档是否来自训练数据集
- 适合关注数据版权和模型透明性的研究者
本文提出Llamazip，一种基于LLaMA3语言模型预测能力的新型无损文本压缩算法。该方法通过仅存储模型无法预测的词元，实现显著的数据压缩，优化存储效率的同时保证数据完整性。论文分析了量化、上下文窗口大小等关键因素对压缩比和计算开销的影响。除压缩外，Llamazip还具备识别文档是否曾用于语言模型训练的能力，有助于解决数据溯源、知识产权和模型训练透明性等核心问题。
原文摘要 · Abstract (English)
This work introduces Llamazip, a novel lossless text compression algorithm based on the predictive capabilities of the LLaMA3 language model. Llamazip achieves significant data reduction by only storing tokens that the model fails to predict, optimizing storage efficiency without compromising data integrity. Key factors affecting its performance, including quantization and context window size, are analyzed, revealing their impact on compression ratios and computational requirements. Beyond compression, Llamazip demonstrates the potential to identify whether a document was part of the training dataset of a language model. This capability addresses critical concerns about data provenance, intellectual property, and transparency in language model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。