arXiv:2502.14444cs.CL2025-02

改进压缩分类算法,提升新闻文章分类准确率

An Enhancement of Jiang, Z., et al.s Compression-Based Classification Algorithm Applied to News Article Categorization

  • 提取词元后压缩,避免全文压缩的局限
  • 压缩效率提升,最长文档分类准确率提高11%
  • 适合高多样性标签和复杂文本的轻量级分类

本研究针对江等人的基于压缩的分类算法在检测文本语义相似性上的不足,提出改进方案。通过优化词元提取与拼接策略,不再依赖整篇文档压缩。压缩提取出的词元可缓解gzip滑动窗口带来的限制,提升压缩效率与相似性检测能力。采用词元并集替代直接拼接,减少冗余,增强归一化压缩距离(NCD)计算的准确性。在不同规模与复杂度的数据集上实验显示,平均准确率提升5.73%,长文档数据集最高提升达11%。该改进在标签多样性高、文本结构复杂的场景下效果更显著。方法保持计算高效,适用于资源受限环境。为文本分类提供了一种高效、可扩展的轻量级预处理解决方案。

原文摘要 · Abstract (English)

This study enhances Jiang et al.'s compression-based classification algorithm by addressing its limitations in detecting semantic similarities between text documents. The proposed improvements focus on unigram extraction and optimized concatenation, eliminating reliance on entire document compression. By compressing extracted unigrams, the algorithm mitigates sliding window limitations inherent to gzip, improving compression efficiency and similarity detection. The optimized concatenation strategy replaces direct concatenation with the union of unigrams, reducing redundancy and enhancing the accuracy of Normalized Compression Distance (NCD) calculations. Experimental results across datasets of varying sizes and complexities demonstrate an average accuracy improvement of 5.73%, with gains of up to 11% on datasets containing longer documents. Notably, these improvements are more pronounced in datasets with high-label diversity and complex text structures. The methodology achieves these results while maintaining computational efficiency, making it suitable for resource-constrained environments. This study provides a robust, scalable solution for text classification, emphasizing lightweight preprocessing techniques to achieve efficient compression, which in turn enables more accurate classification.

文本分类压缩算法轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。