提出一种检测大模型文本分布偏移的不对称近邻测试方法。
ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language
- 基于有向k近邻图构建不对称双样本检验，区分提示与生成文本
- 在真实数据中有效识别大模型幻觉和人工文本的分布偏差
- 计算高效易实现，适合从隐藏状态检测语言模型输出异常
大模型生成内容中的幻觉和人工文本常表现为提示与响应隐藏状态分布的偏离。由于提示或检索上下文通常作为参考样本，而响应作为查询样本，两者长度差异显著，这种不对称性促使我们采用对两类样本区别对待的统计检验方法。本文提出基于有向k近邻图的不对称两样本检验ASK-NN，其统计量计算的是在合并样本中最近邻仍为参考样本的参考点数量。在置换零假设下，该统计量具有精确的有限样本条件均值与方差；进一步证明了在固定替代假设下的渐近正态性和一致性。ASK-NN计算高效且易于实现。实验表明，它在合成基准、人工文本检测以及基于词元级隐藏状态的大模型幻觉检测任务上，性能可媲美核方法与图基基线。
原文摘要 · Abstract (English)
Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions. Since prompts or retrieved contexts typically serve as reference samples and responses as query samples, with major differences in length, these asymmetries motivate the use of change test statistics that treat the two samples differently. We consider an asymmetric two-sample test ASK-NN based on the directed k-nearest-neighbor graph. Our statistic counts reference points whose nearest neighbor in the pooled sample is also a reference point. Under the permutation null, it admits an exact finite-sample conditional mean and variance; we further establish asymptotic normality and consistency under fixed alternatives. ASK-NN is computationally effective and easy to implement. Empirically, it is competitive with kernel and graph-based baselines on synthetic benchmarks, artificial-text detection, and LLM hallucination detection from token-level hidden states.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。