用读者划重点数据评估模型压缩效果，避免位置和长度偏差影响。
Measuring Alignment With Reader Highlights Net of Position and Length
- 通过匹配相同位置深度和长度排名的句子，消除人为划重点的偏倚。
- 语言模型保留38.4%的读者标记句，显著优于随机水平（p=0.0005）。
- 适合关注模型可解释性与文本摘要公平评估的研究者。
上下文压缩会舍弃文档大部分内容，传统评估依赖下游任务准确率，形成评价循环。自然社交性划重点提供非循环参照：多人独立标记同一页内容。但简单采用保留的标记句比例会受双重干扰——标记句集中在前部且更长，导致任何偏好开头或长句的方法得分虚高。本文通过将每句标记句与同文档中相对位置和长度排名相同的未标记句配对，并在合成零假设样本上校准所有估计器，解决此问题。在120篇网页文档（每篇至少12位独立读者）上，语言模型重要性排序保留了38.4%的标记句，相比其匹配对照组（19.9%）提升+0.196 [+0.148, +0.239]，在精确随机化检验下显著（p=0.0005），跨厂商复现。朴素截断法（仅按位置）仅得+0.003。对比可见，单个真人读者达到+0.182，与GPT-5.4相近；而经典方法如Luhn（1958）达+0.088，表明部分共识可通过词频恢复，但引入词汇中心性后仅降0.010，说明共识不主要源于语义核心。此外，本文发现先前工作中的一个结论在该数据集上无法复现。
原文摘要 · Abstract (English)
Context compression discards most of a document before a language model reads it, and is normally evaluated by downstream task accuracy - which makes another model the judge of what mattered. Naturalistic social highlighting offers a non-circular reference: many people independently marking passages on the same page. But the obvious metric, the fraction of crowd-marked sentences a compressor keeps, is confounded twice: crowd marks are front-loaded and crowd-marked sentences are longer, so any method favouring early or long sentences scores well regardless of readers. We remove both by matching each marked sentence against unmarked sentences of the same document at equal relative depth and equal within-document length rank, and we calibrate every estimator on synthetic nulls built from position and length alone - a step that matters, since depth-only stratification returns a false positive on 20-36% of nulls containing no effect. On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours: an enrichment of +0.196 [+0.148, +0.239], at p = 0.0005 under an exact randomization test that assumes nothing about clustering, and replicated cross-vendor. Naive truncation, whose keep rule is position, correctly falls to +0.003. To give the number a scale: scored identically, on the same budget, against a crowd label recomputed to exclude them, a single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below Claude Opus 5. Classical methods are not null - Luhn's 1958 heuristic reaches +0.088 - so reader selection is partly recoverable by counting words; conditioning additionally on lexical centrality removes only 0.010, so the agreement is not centrality. We also report that a claim in our own prior work does not reproduce on this corpus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。