token计数不能作为模型家族溯源的唯一依据
Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting
- 设计有效性约束的验证协议,区分有效差异与数据缺失导致的无效结果
- 仅6组测试对在保留集上达标,准确率75%,特异性100%
- 证实token计数可作分词栈指纹,但不足以判断模型家族关系
当大语言模型通过中继或转售API提供服务时,黑箱模型溯源日益重要。一个低成本信号是OpenAI兼容接口返回的提示词计数:共享分词器和对话模板的模型可能产生相同的计数序列(固定偏移)。然而该信号在更广范围的模型家族归属中的有效性尚未经过直接保留测试。本研究对24个已标注端点对进行冻结阈值实验,均分为开发集与未触碰保留集,每对30个受控文本,重复3次。引入有效性约束的结果合约,以区分真实差异与因缺失使用数据、速率限制或端点策略导致的无信息测量。所得不变位精确匹配得分完美区分12组开发对,得出冻结阈值0.725。在保留集上,仅6组满足预设三重重复规则。在可评估对中,平衡准确率为0.75,敏感性为0.50(95%威尔逊区间0.15–0.85),特异性为1.00(0.342–1.00)。两组同家族对(Qwen 3.8与DeepSeek V4变体)低于冻结阈值。4,320次正式API调用全部可重放,保留集中存在189次非200响应及157次成功响应无提示词使用记录。研究验证了计数一致性作为共享分词栈指纹的有效性,但否定了其作为模型家族血缘关系独立必要检验的适用性。
原文摘要 · Abstract (English)
Black-box model attribution is increasingly relevant when large language models (LLMs) are served through relay and reseller APIs. A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint: two models that share a tokenizer and chat template may produce the same count sequence up to a fixed offset. Yet the validity of this signal for broader \emph{model-family} attribution has received little direct holdout testing. We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set, with three temporal repeats and 30 controlled texts per pair. We introduce a validity-gated result contract that distinguishes an observed dissimilarity from an uninformative measurement caused by missing usage data, rate limits, or endpoint policy. The resulting shift-invariant exact-match score perfectly separates the 12 development pairs, yielding a frozen threshold of 0.725. On holdout, however, only 6 of 12 pairs are eligible under the pre-specified three-repeat rule. Among eligible pairs, balanced accuracy is 0.75, sensitivity is 0.50 (95\% Wilson interval 0.15--0.85), and specificity is 1.00 (0.342--1.00). Two same-family pairs---Qwen 3.8 and DeepSeek V4 variants---fall below the frozen threshold. Across 4,320 formal API calls, every log is replayable, while holdout contains 189 non-200 responses and 157 successful responses without prompt-token usage. The study therefore validates token-count consistency as a fingerprint of a shared \emph{tokenization stack}, but rejects its use as a standalone necessary test for model-family lineage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。