模仿人类略读策略，提升长文档表示效率与效果
Skim-Aware Contrastive Learning for Efficient Document Representation
- 通过随机遮蔽段落并利用NLI对比目标对齐相关部分
- 在法律和生物医学数据集上准确率显著提升，计算效率更高
- 适合需要高效理解长文本的法律、医疗等专业领域
尽管基于Transformer的模型在词级和句级任务中表现优异，但在法律、医学等领域的长文档表示仍具挑战。稀疏注意力机制虽可处理更长输入，但资源消耗大且难以捕捉全文上下文；层次化Transformer模型效率较高，却未能清晰说明文档各部分间的关联。受人类略读行为启发，我们提出一种新的自监督对比学习框架：随机遮蔽文档一段，利用自然语言推理（NLI）构建对比目标，使被遮蔽段落与相关内容对齐，与无关内容分离。该方法模拟人类信息整合方式，生成更具语义丰富性且计算更高效的文档表示。在法律与生物医学文本上的实验表明，该方法在准确率和效率方面均有显著提升。
原文摘要 · Abstract (English)
Although transformer-based models have shown strong performance in word- and sentence-level tasks, effectively representing long documents, especially in fields like law and medicine, remains difficult. Sparse attention mechanisms can handle longer inputs, but are resource-intensive and often fail to capture full-document context. Hierarchical transformer models offer better efficiency but do not clearly explain how they relate different sections of a document. In contrast, humans often skim texts, focusing on important sections to understand the overall message. Drawing from this human strategy, we introduce a new self-supervised contrastive learning framework that enhances long document representation. Our method randomly masks a section of the document and uses a natural language inference (NLI)-based contrastive objective to align it with relevant parts while distancing it from unrelated ones. This mimics how humans synthesize information, resulting in representations that are both richer and more computationally efficient. Experiments on legal and biomedical texts confirm significant gains in both accuracy and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。