用1400个不可压缩的事实题，估算大模型的真实参数量。
Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
- 设计1400道无法通过推理或架构优化压缩的事实题，构建不可压缩知识探针。
- 通过准确率与参数量的对数线性关系，实现93个开源模型上91%的拟合度。
- 适用于评估闭源大模型的容量等级，尤其适合对比不同厂商模型的能力差距。
闭源前沿模型不公开参数量。存储F条事实至少需要F/(每参数位数)个权重，因此事实回忆率可作为参数量的下界——一种内在且与服务无关的信号。本文提出不可压缩知识探针（IKP），包含1400个跨越7个隐蔽层级的事实问题，旨在隔离无法通过推理推导或架构改进压缩的知识。评分时不施加幻觉惩罚（λ=0），仅以正确回答比例衡量精度，避免超参数和分层阈值选择的影响；全范围λ×地板值消融实验表明校准结果在不同评分策略下仍稳健，而个体估计则敏感，支持无惩罚默认设定。在93个开源模型（135M–1,600B，19个供应商）上校准对数线性映射，获得R²=0.910；留一法交叉验证确认泛化能力（中位折叠误差1.48倍，72%在2倍内，86%在3倍内）。该工具故意粗略——90%预测区间跨度达±3倍，宽于推理经济考量，故仅能恢复数量级有效容量与相对排名，而非精确参数数。对混合专家模型，总参数量比活跃参数量更能预测知识（R²=0.67 vs 0.41）。在筛选后的1311个探针上评估201个模型（27个供应商），报告所有主流闭源前沿模型的有效知识容量为预测带而非点估计；对高度安全调优模型，此值为下界，因拒绝策略可能抑制数十个百分点的本可回答容量。
原文摘要 · Abstract (English)
Closed-source frontier labs do not disclose parameter counts. Storing F facts requires at least F/(bits per parameter) weights, so factual recall lower-bounds parameter count--an intrinsic, serving-independent signal, though (as we show) a coarse one. We introduce Incompressible Knowledge Probes (IKPs), a benchmark of 1,400 factual questions spanning 7 tiers of obscurity, designed to isolate knowledge that cannot be derived by reasoning or compressed by architectural improvements. We score with no hallucination penalty (lambda = 0: IKP accuracy is simply the fraction of probed facts answered correctly), which removes both the penalty hyperparameter and the per-tier flooring choice; a full lambda x flooring ablation shows the calibration is robust across scoring choices while individual estimates are not, motivating the no-penalty default. We calibrate a log-linear mapping from IKP accuracy to parameter count on 93 open-weight models (135M-1,600B) spanning 19 vendors, achieving R^2 = 0.910; leave-one-out cross-validation confirms generalization (median fold error 1.48x, 72% within 2x, 86% within 3x). The instrument is deliberately coarse--its 90% prediction interval spans ~3x in either direction, wider than inference economics--so IKP recovers order-of-magnitude effective capacity and relative rankings, not precise parameter counts. For Mixture-of-Experts models, total parameters predict knowledge (R^2 = 0.67) better than active parameters (R^2 = 0.41). We evaluate 201 models from 27 vendors on a curated probe set (1,311 of 1,400 probes surviving name-collision and label-ambiguity filters) and report effective knowledge capacity for all major proprietary frontier models as prediction bands rather than point estimates; for heavily safety-tuned models these are lower bounds, since refusal policy can suppress tens of percentage points of otherwise-answerable capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。