测试压缩提示对扩散模型效果，发现语义保留不等于推理稳定。
Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA
- 用LLMLingua-2压缩提示，测试在扩散模型上的表现。
- 数学推理任务准确率大幅下降，但摘要任务仍稳健。
- 压缩失败主因是关键信息遗漏，非语义偏差，适合扩散模型研究者。
提示压缩可降低大模型推理开销和上下文长度，但以往评估多集中于自回归架构。本研究检验LLMLingua-2在扩散大语言模型（DLLMs）LLaDA-8B-Instruct上的迁移效果。在GSM8K、DUC2004和ShareGPT三个数据集上，各使用250个提示，在约50%压缩率下评估数学推理、提示重建和摘要任务。通过精确匹配、BLEU、ROUGE和BERTScore对比原始、压缩及重构提示的输出。结果表明，高语义保真度并不保证扩散模型下游行为稳定：摘要任务相对鲁棒，而数学推理虽保持高语义相似性却显著退化。重构分析显示，语义相近提示可能遗漏影响去噪稳定的推理关键信息。整体来看，压缩失败主要源于信息缺失而非语义漂移，说明自回归提示压缩方法难以直接适用于DLLMs。研究呼吁开发面向扩散模型的压缩策略。
原文摘要 · Abstract (English)
Prompt compression reduces inference cost and context length in large language models, but prior evaluations focus mainly on autoregressive architectures. This study examines whether LLMLingua-2 transfers effectively to diffusion large language models (DLLMs), specifically LLaDA-8B-Instruct. We evaluate GSM8K, DUC2004, and ShareGPT using 250 prompts per dataset at an approximate 50\% compression ratio, covering mathematical reasoning, prompt reconstruction, and summarization. Outputs from original, compressed, and reconstructed prompts are compared using exact-match accuracy, BLEU, ROUGE, and BERTScore. Results show that high semantic preservation does not necessarily ensure stable downstream behavior in diffusion models. Summarization remains relatively robust, while mathematical reasoning degrades substantially despite high semantic similarity. Reconstruction further shows that semantically similar prompts may omit reasoning-critical information needed for stable denoising. Overall, compression failures are mainly driven by information omission rather than semantic drift, suggesting that autoregressive prompt compression methods may not transfer uniformly to DLLMs. These findings motivate diffusion-aware compression strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。