用单次前向传播计算文本多样性,无需额外模型或人工标注。
"I've Seen How This Goes": Characterizing Diversity via Progressive Conditional Surprise

- 基于上下文学习和信息论,通过模型输出概率直接评分
- 在人类基准上达0.846的OCA,接近最强基线
- 可检测训练中创意性下降,适合评估生成质量
衡量创造性输出的多样性是评估训练后模式崩溃、比较解码策略以及量化人工智能与人类写作创造力的核心问题。本文提出一种新方法:利用上下文学习,通过单次前向传播读取基础模型θ在每种排列下的逐标记对数概率,得到无嵌入模型、无参考语料库、无人工标签的字节级得分——即“Decan”指标$D_{Ca_n} = C \times a_n$。该方法基于信息论,借助语言模型的上下文学习能力,检测任意数量输入间的广泛相似性,无需训练专用模型。同一流程可同时评估AI样本与人类写作,将多样性视为(响应、提示、评分模型)的属性。在Tevet与Berant的人类基准McDiv上,$D_{Ca_n}$在prompt_gen集上达到OCA 0.846,略低于最强神经基线SentBERT(0.897)。在OLMo-2-7B的训练流程中,$D_{Ca_n}$从基础模型→SFT→DPO→RLVR阶段持续下降,准确捕捉了创意写作关注的多样性损失。
原文摘要 · Abstract (English)
Measuring the diversity of creative outputs is central to evaluating post-training mode collapse, comparing decoding strategies, and quantifying creative behavior in both AI and human writing. We propose a new approach to measuring diversity using in-context learning, of which the ``Decan'' metric, $D_{Ca_n} = C \times a_n$, is the working instance we evaluate: a per-byte score read off the per-token log-probabilities of a base model $θ$ in a \emph{single forward pass} per permutation, with no embedding model, no reference corpus, and no human labels. This approach is grounded in information theory, makes use of language model in-context learning to detect a wide range of similarities between any number of inputs, and obviates the need to train a special-purpose model. The same pipeline scores AI samples and human-written response sets, with diversity treated as a property of (responses, prompt, scoring model). On Tevet and Berant's human-grounded McDiv benchmark, $D_{Ca_n}$ reaches OCA 0.846 on the McDiv prompt\_gen set where it performs best, behind the strongest neural baseline reported in Tevet and Berant (SentBERT, 0.897). On the OLMo-2-7B post-training pipeline, $D_{Ca_n}$ drops monotonically across the base $\to$ SFT $\to$ DPO $\to$ RLVR stages, detecting the type of diversity loss that creative-writing applications care about.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。