论文发现:文档结构标记比格式细节更能影响模型表现。
The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
- 提出'干净窗口存活率'量化文本边界依赖度,评估十三个语料库
- 删除结构提示使文本预测难度显著上升,但更换标记符号无影响
- 建议用纯框架格式训练,保留可逆侧车文件,记录提取器信息
文档的标注方式是未被数据卡记录的训练变量。现有研究证实文本抽取选择会改变模型行为,但从未测量这些选择对语料库标注的影响。本文定义了‘干净窗口存活率’,即流中仍需边界推理的部分的确定性计数,并从三方面进行测量:语料库承载能力——对十三个公开语料库的普查显示,视觉转换的PDF片段存活率仅0.153,而C4为0.889;真正稀缺的不是无标记文本,而是长段无标记文本;预注册供应测试表明剩余内容属机构而非消费来源。我们自己的预注册预测失败:转换器不会在散文中虚构结构,这一零结果迫使建立可靠机制以维持其有效性。读者使用情况——在五种基础模型(0.6B至8.2B参数)和两条流水线中,删除结构公告使后续散文预测显著变难,但更换标记符号无影响。零结果并不意味着标注不重要,而是将变量转移:有效线索是公告本身,而非符号。作者施加影响——设定有界零假设。基础模型不会在原文基础上强制添加标记体,即使删除所有公告,它们也不会自行恢复,其概率与零无异,且与人工基准一致。我们提供测量所暗示的格式:纯框架,按原始顺序排列段落,所有公告删除并置于可逆侧车中,通过公告存在与否而非标记方式混合对比。应根据训练能力而非保真度选择格式操作符,并在数据卡上记录提取器身份与存活率。
原文摘要 · Abstract (English)
How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。