在边缘设备上实现低延迟智能助手，靠分时记忆管理提升体验
Embedded AI Companion System on Edge Devices
- 用活跃/休眠交替机制，动态分配计算资源
- 7B量化模型比无记忆大模型表现更好，接近GPT-3.5效果
- 专设评测基准，兼顾对话质量与长期记忆能力
边缘设备的计算资源限制使得构建具备良好用户体验的全嵌入式AI助手困难重重。现有文献中的AI助手与记忆系统因计算资源不足和延迟问题，难以直接部署。本文提出一种新型记忆范式：用户活跃时，系统通过轻量检索实现低延迟实时对话；用户空闲时，则执行耗时的记忆提取、整合与维护任务。该设计在严苛的硬件约束下，既保持低延迟，又实现长期个性化。我们还构建了一个综合评测基准，全面评估助手的对话质量与记忆能力。实验表明，使用极弱模型（Qwen2.5-7B-Instruct int4量化）的系统，在多数指标上优于无记忆的原始大模型，并达到与16k上下文窗口的GPT-3.5相当的性能。
原文摘要 · Abstract (English)
Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。