用4比特小模型在设备端压缩提示词，省下大量云端推理成本。
Closing the Social-Semantic Gap: SPSD for Edge-Based Prompt Compression in Cloud LLM Inference
- 用4比特量化小模型在边缘端压缩用户提示，减少传输数据量。
- 每轮压缩平均节省99.9个输入词元，146次实验全为正向节省。
- 压缩后回答质量与原版无显著差异，适合对成本敏感的部署场景。
大型语言模型（LLM）推理中的预填充阶段正成为云规模能源消耗的重要来源。许多面向消费者和对话类提示包含社会性支架：礼貌用语、道歉前言、重复和关系建立语言，这些对人类交流重要但对机器推理信息量低。我们称此差异为“社会-语义鸿沟”。本文提出SPSD（情感保持语义蒸馏），一种基于边缘端的提示压缩流水线，利用4比特量化的小型语言模型（SLM）在传输至云端大模型前压缩用户提示。在248条提示语料上评估，使用Gemma-2-2B-Instruct（Q4_K_M）作为SLM，Llama-3.1-8B-Instruct作为云端评估模型，每轮压缩平均节省99.9个输入词元，146次实验均实现正向节省。通过盲评方式，由大模型作为评判者对121组输出评分，在15分制中压缩路径与原始路径差距在1分以内，43%为平局，28%压缩胜出，29%原始胜出。余弦相似度中位数为0.712，均值0.682，54.1%的对比对高于0.70阈值。安全关键领域通过规则门控保守走直通路径。在给定假设下，每轮可节省70–270微瓦时能量。SPSD表明，设备端提示蒸馏可在不损失响应质量的前提下，显著降低云端大模型输入词元开销。
原文摘要 · Abstract (English)
The prefill stage of Large Language Model (LLM) inference is a growing contributor to cloud-scale energy cost. Many consumer-support and conversational prompts contain social scaffolding: politeness markers, apologetic preamble, repetition, and rapport-building language that is important for human communication but carries low marginal information for machine reasoning. We call this discrepancy the Social-Semantic Gap. We present SPSD (Sentiment Preserving Semantic Distillation), an edge-based pipeline that compresses user prompts using a 4-bit quantised Small Language Model before transmission to a cloud-deployed LLM. Evaluation on a 248-prompt corpus using Gemma-2-2B-Instruct (Q4_K_M) as the SLM and Llama-3.1-8B-Instruct as the cloud evaluation model yields a mean input token saving of 99.9 tokens per distilled call, with all 146 distilled calls yielding positive savings. Response quality, assessed by blind LLM-as-judge scoring across 121 pairs, is non-inferior to the raw path within a pre-specified 1-point margin on a 15-point rubric; the judge awarded 43 percent ties, 28 percent distilled wins, and 29 percent raw wins. Cosine similarity is mixed: mean 0.682, median 0.712, with 54.1 percent of pairs above the 0.70 reference threshold. Safety-critical domains are conservatively routed to passthrough via rule-based gates. Per-call net energy saving is estimated at 70-270 uWh under stated assumptions. SPSD shows that on-device prompt distillation can reduce cloud LLM input-token cost while preserving response quality within a practical non-inferiority margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。