让AI提前理解文本走向，提升生成内容的连贯性。
Look Ahead Text Understanding and LLM Stitching
- 用BERT与GPT融合方法预测文本发展方向
- 在含噪声文本上性能超越现有模型
- 适合做社交媒体内容预判与智能对话系统
本文提出「前瞻文本理解」问题，并以前瞻段落识别（LASI）为例。该问题在生成式AI和人机交互中普遍存在，旨在理解正在发展中的文本或对话的潜在方向。我们采用基于Transformer的大型语言模型解决此问题，发现LASI比经典段落识别更难。我们认为，双向上下文信息（如BERT）和单向预测能力（如GPT）均有助于任务。为此，我们提出两种融合BERT与GPT的方法。实验表明，所提方法在含噪声文本（常见于生成式AI的动态文本）中表现更优，显著超越现有模型。本研究为其他前瞻文本理解任务（如前瞻情感分类）提供思路，指出了通过模型拼接利用预训练LLM的潜力。
原文摘要 · Abstract (English)
This paper proposes a look ahead text understanding problem with look ahead section identification (LASI) as an example. This problem may appear in generative AI as well as human interactions, where we want to understand the direction of a developing text or conversation. We tackle the problem using transformer-based LLMs. We show that LASI is more challenging than classic section identification (SI). We argue that both bidirectional contextual information (e.g., BERT) and unidirectional predictive ability (e.g., GPT) will benefit the task. We propose two approaches to stitch together BERT and GPT. Experiments show that our approach outperforms the established models, especially when there is noise in the text (which is often the case for developing text in generative AI). Our paper sheds light on other look ahead text understanding tasks that are important to social media, such as look ahead sentiment classification, and points out the opportunities to leverage pre-trained LLMs through stitching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。