通过缓解过平滑和引入异常评分，提升轻量级长文本压缩效果
MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores
- 用类别间余弦相似性损失缓解令牌表示过平滑问题
- 在4倍压缩比下实现3.3倍推理加速，移动端表现优异
- 适合资源受限设备上的长文本处理任务
大语言模型在处理长上下文输入方面取得显著进展，但在资源受限环境中的推理时间与资源消耗仍成挑战。为此，我们提出MOOSComp，一种基于令牌分类的长文本压缩方法，通过缓解过平滑问题并引入异常评分来增强BERT基压缩器性能。训练阶段添加类别间余弦相似性损失项，惩罚过度相似的令牌表示，从而提升分类准确率；压缩阶段引入异常评分，保留易被忽略的关键稀有令牌。该评分与分类器输出融合，使压缩器对多种任务更具泛化能力。在多个压缩比下，长文本理解与推理基准测试均表现更优。尤其在资源受限移动设备上，4倍压缩比下实现3.3倍速度提升。
原文摘要 · Abstract (English)
Recent advances in large language models have significantly improved their ability to process long-context input, but practical applications are challenged by increased inference time and resource consumption, particularly in resource-constrained environments. To address these challenges, we propose MOOSComp, a token-classification-based long-context compression method that enhances the performance of a BERT-based compressor by mitigating the over-smoothing problem and incorporating outlier scores. In the training phase, we add an inter-class cosine similarity loss term to penalize excessively similar token representations, thereby improving the token classification accuracy. During the compression phase, we introduce outlier scores to preserve rare but critical tokens that are prone to be discarded in task-agnostic compression. These scores are integrated with the classifier's output, making the compressor more generalizable to various tasks. Superior performance is achieved at various compression ratios on long-context understanding and reasoning benchmarks. Moreover, our method obtains a speedup of 3.3x at a 4x compression ratio on a resource-constrained mobile device.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。