arXiv:2608.16391cs.CRcs.AI2026-08

提出新方法检测第三方大模型API的推理质量,尤其关注长任务中的极端错误。

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

  • 通过多次请求重建输出分布,用平均保真度损失评估整体可靠性。
  • 在七次路由快照中发现不同路由存在显著的极端保真度损失差异。
  • 适合评估长时序智能体任务,对高阶推理错误更敏感。

随着大语言模型广泛应用,第三方部署的开源权重模型已成为重要生态组成部分。本文将托管模型路由形式化为随机过程,提出 extbf{Ventor-QTest},一种无需目标API概率信息的复合黑盒审计方法。其重复请求组件将每个固定约束上下文发送多次,从返回文本统计重建类别输出分布,报告经无偏修正的窗口内均值粗粒化KL散度(平均保真度损失,AFL)。其长序列组件通过独立运行报告运行级参考中心惊喜度的实证上尾值(极端保真度损失,EFL)。在三种支持logprob的路由条件下,AFL与基于logprob的粗粒化KL对比显示强线性拟合。在七次路由快照中,20次序列探测揭示路由特异性的EFL变化。AFL与EFL与GPQA-Diamond准确率无显著路由级关联。相反,显著的EFL与终端任务通过率随任务暴露增加而下降现象一致。该模式可能源于长时序任务对极端保真度损失更敏感。结果表明应联合报告AFL与EFL,尤其在审计长时序智能体任务时。开源实现已发布于https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest。

原文摘要 · Abstract (English)

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.

大模型审计API验证保真度损失长序列任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。