通过多层级视觉聚焦机制，提升无监督文本生成指令的多样性与难度
Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation
- 借鉴人类视觉层级结构，分三个粒度提取文本信息
- 在多个数据集上合成指令的多样性和难度显著优于现有方法
- 适合需要高质量指令数据的LLM训练场景
从无监督文本中合成高质量指令数据是训练大语言模型的一种有前景范式，但现有自动化方法在生成指令的多样性和难度方面仍存在明显不足。为此，我们提出Self-Foveate，一种基于大语言模型的指令合成方法。受人类视觉感知层级启发，Self-Foveate引入“微观-散点-宏观”多层级聚焦机制，从细粒度细节、跨区域关联到整体模式三个互补层次引导文本信息提取，从而提升合成指令的多样性和难度。此外，还设计了重合成模块，以增强指令对源文本的保真度和整体质量。在多个无监督语料库和不同模型架构上的综合实验表明，Self-Foveate始终优于现有方法。代码已公开：https://github.com/Mubuky/Self-Foveate
原文摘要 · Abstract (English)
Synthesizing high-quality instruction data from unsupervised text is a promising paradigm for training large language models (LLMs), yet automated methods for this task still exhibit significant limitations in the diversity and difficulty of synthesized instructions. To address these challenges, we propose Self-Foveate, an LLM-driven method for instruction synthesis. Inspired by hierarchical human visual perception, Self-Foveate introduces a "Micro-Scatter-Macro" multi-level foveation methodology that guides the extraction of textual information at three complementary granularities, from fine-grained details through cross-region connections to holistic patterns, thereby enhancing both the diversity and difficulty of synthesized instructions. Furthermore, a re-synthesis module is incorporated to improve the fidelity of instructions to source text and their overall quality. Comprehensive experiments across multiple unsupervised corpora and diverse model architectures demonstrate that Self-Foveate consistently outperforms existing methods. We publicly release our code at https://github.com/Mubuky/Self-Foveate
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。