用软提示复用推测解码模块，实现高效高精度的实时分类
Speculative Probing: LLM Monitoring at Speculative-Decoding Cost
- 在目标序列末尾加训练好的软提示，复用推测解码模块作分类器
- 四模型四任务测试中，小模型性能超零样本GPT-5.4-mini，媲美8B专用安全模型
- 无需运行完整大模型，推理开销几乎为零，适合实时监控场景
语言模型推理时的实时分类对安全过滤、行为分析和模型监控至关重要，但现有方法在准确率与效率间存在固有权衡。隐状态探测虽快，却或缺乏上下文感知（仅处理单个向量），或代价高昂（需专用分类模型或对所有标记计算并聚合）。本文发现，近期大模型中的推测解码模块可被重用于高效高质量分类：通过在目标序列末尾添加训练好的软提示，将推测解码模块转变为序列分类器。推理时，由于KV缓存已在GPU内存中，分类几乎无额外开销。在四个模型（Qwen3.5-4B、9B、27B，MiniCPM4.1-8B）上的四个分类任务上评估，小型探测器持续优于零样本GPT-5.4-mini，且在多语言提示安全任务上达到或超越专用8B安全分类器（Qwen3Guard-Gen-8B，Llama-Guard-3-8B），而无需运行完整LLM。
原文摘要 · Abstract (English)
Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring, but current approaches force a trade-off between accuracy and efficiency. Hidden-state probes are fast but limited: they are either not context-aware: operating on a single vector and cannot model interactions across positions; or they are very costly: having dedicated classifier models (Llama Guard, Qwen Guard, LLM-as-judge) or performing computation on hidden states for all tokens and then pooling the results (MultiMax). This shows an intrinsic trade-off between efficiency and accuracy. However, we find that the speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification. By appending a trained soft prompt at the end of the target sequence, we can repurpose the speculative-decoding module into a sequence classifier. At inference time in a speculative-decoding pipeline, the KV cache is already in GPU memory, so classification adds negligible overhead. We evaluate on four classification tasks across four models (Qwen3.5-4B, 9B, 27B, MiniCPM4.1-8B). Our small probes consistently outperform zero-shot GPT-5.4-mini and, on multilingual prompt safety, match or beat specialized 8B safety classifiers (Qwen3Guard-Gen-8B, Llama-Guard-3-8B) without running a full LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。