通过分层上下文分析文本概率,提升模型训练数据归属与生成文本检测精度。
Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated Text Detection

- 构建嵌套左文窗口的似然数组,捕捉不同上下文尺度下的生成特征。
- 在多个模型上显著优于基线方法,尤其在短上下文信息利用上表现突出。
- 适合关注隐私泄露与内容真实性验证的研究者使用。
成员推理旨在判断某文本是否用于训练语言模型,而生成文本检测则判断文本是否由模型生成而非人类撰写。现有基于似然的方法通常将词元级概率压缩为少数预设得分,且主要依赖完整前序上下文的概率。本文提出似然数组回归(LAR),评估目标词元在嵌套左文窗口下的条件似然,并将所得特征组织为结构化数组。通过跨不同长度文本对齐数组,LAR学习检测信息随上下文尺度、词元位置及似然特征的变化规律。LAR-1 聚合各对齐单元的贡献,LAR-2 引入同一目标词元在不同上下文长度下评估结果的二阶特征。针对路径内二次模型,我们建立了匹配的极小极大下界与上界,刻画了有限维近似和随机平方投影带来的误差,并推导出何种条件下谱筛可达到极小极大率。在多个评分语言模型上,LAR 显著优于基线方法,在成员推理与生成文本检测任务中均有提升。分析表明,短上下文似然包含超越传统全上下文概率的信息,而二阶特征对成员推理带来额外增益。
原文摘要 · Abstract (English)
Membership inference asks whether a text was used to train a language model, whereas AI-generated text detection asks whether it was generated by a language model rather than written by a human. Existing likelihood-based methods typically compress token-level probabilities into a few prespecified scores, most often using only probabilities conditioned on the full preceding context. We propose likelihood-array regression (LAR), which evaluates each target token under nested left-context windows and organizes the resulting likelihood-derived features into a structured array. After aligning arrays across texts of different lengths, LAR learns how detection information varies with context scale, token position, and likelihood features. LAR-1 aggregates learned contributions from individual aligned cells, while LAR-2 adds second-order features formed from pairs of evaluations of the same target token across context lengths. For within-path quadratic model, we establish matching minimax lower and upper bounds, characterize errors from finite-dimensional approximation and random squared projections, and derive conditions under which an oracle spectral sieve attains the minimax rate. Across multiple scoring language models, LAR substantially improves membership inference and AI-generated text detection over likelihood-based baselines. The analyses further show that shorter-context likelihoods contain information beyond conventional full-context probabilities, while second-order features provide additional gains for membership inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。