无需预设问题，一键挖掘语言模型隐含知识图谱。
DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models
- 用低差异序列生成实现无状态、可并行的确定性解码。
- 4比特激活感知量化使概念覆盖率提升30%-170%，而均匀量化导致覆盖率下降71%-86%。
- 适合评估压缩模型的知识广度与事实准确性，尤其关注部署前的可靠性。
现有知识探测方法依赖预定义查询，限制了对已知概念的提取。我们提出DecompressionLM，一种无状态的零样本概念图提取框架，可在不使用预设查询或跨序列共享状态的情况下发现语言模型编码的内容。该方法解决三大问题：(i) 跨序列耦合导致概率集中在高频前缀；(ii) 竞争解码抑制长尾概念；(iii) 顺序探索带来的可扩展性限制。通过结合范德科普特低差异序列与算术解码，DecompressionLM实现无共享状态的确定性、可轻松并行生成。在两种模型家族和五个量化变体中测试发现，激活感知量化（AWQ-4bit）使概念覆盖率提升30%-170%，而均匀量化（GPTQ-Int4）引发71%-86%的覆盖率崩溃——这种差异无法通过解释级困惑度可靠反映。基于语料库的验证进一步揭示，MMLU-Pro Law排行榜顶端与底部模型之间存在19.6分的幻觉差距。DecompressionLM将概念覆盖率确立为评估压缩模型知识广度与事实根基性的补充评价维度。
原文摘要 · Abstract (English)
Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: (i) cross-sequence coupling that concentrates probability mass on high-frequency prefixes, (ii) competitive decoding effects that suppress long-tail concepts, and (iii) scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse - divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 19.6-point hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models intended for deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。