动态构建语义聚合向量，提升大模型推理时的任务适应能力
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations
- 通过全查询旋转策略提取稳定语义向量，减少上下文学习干扰
- 根据任务复杂度动态分割向量，并优化注入位置，性能超越少样本ICL
- 轻量高效，适合低资源场景下的推理时任务调整
基于上下文的向量（ICV）方法从大语言模型中提取任务相关表示并在推理时重注入，实现与少样本上下文学习（ICL）相当的效果，且无需重复处理演示。然而现有ICV方法仍受ICL特定因素影响，常使用粗粒度或语义碎片化的表示作为向量源，并依赖启发式注入位置，限制了适用性。为此，本文提出动态向量（DyVec），采用全查询旋转（EQR）策略，通过缓解ICL引入的方差，提取稳健的语义聚合潜在表示；随后利用动态潜在分割与注入，根据任务复杂度自适应划分表示，并通过基于REINFORCE的优化学习每一段的最佳注入位置。实验结果表明，DyVec在多个任务上优于少样本ICL、LoRA及先前的ICV基线。进一步分析验证了动态分割和注入语义聚合表示的有效性。DyVec为推理时任务适配提供了一种轻量且数据高效的解决方案。
原文摘要 · Abstract (English)
In-Context derived Vector (ICV) methods extract task-relevant representations from large language models (LLMs) and reinject them during inference, achieving comparable performance to few-shot In-Context Learning (ICL) without repeated demonstration processing. However, existing ICV methods remain sensitive to ICL-specific factors, often use coarse or semantically fragmented representations as the source of the vector, and rely on heuristic-based injection positions, limiting their applicability. To address these issues, we propose Dynamic Vector (DyVec), which incorporates an Exhaustive Query Rotation (EQR) strategy to extract robust semantically aggregated latent representations by mitigating variance introduced by ICL. It then applies Dynamic Latent Segmentation and Injection to adaptively partition representations based on task complexity and leverages REINFORCE-based optimization to learn optimal injection positions for each segment. Experiments results show that DyVec outperforms few-shot ICL, LoRA, and prior ICV baselines. Further analysis highlights the effectiveness of dynamically segmenting and injecting semantically aggregated latent representations. DyVec provides a lightweight and data-efficient solution for inference-time task adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。