Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection
Researchers introduced PRECOG (Pre-Computed Context Injection), a retrieval mechanism for State-Space Models (SSMs) that pre-encodes document corpora offline as fixed-size hidden states and injects the best-matching state at query time, reducing prefill cost from O(L_context) to O(1). The same state-injection mechanism enables SMC (Structured Memory Consolidation), a hierarchical persistent memory with cognitive-domain clustering, adjustable fidelity-vs-storage, and O(1) session initialization. The system was demonstrated on TENNs-LLM, a 1.2B-parameter gated-SSM language model with a 192 KB hidden state.
A new paper proposes PRECOG and SMC, leveraging the fixed-size, position-agnostic recurrent hidden state of SSMs to achieve O(1) retrieval and persistent memory for edge language models. The approach eliminates the prefill cost of RAG and the growing KV-cache of Transformers, demonstrated on a 1.2B-parameter SSM.
The key innovation is exploiting the SSM's hidden state as a complete, fixed-size summary of all processed text, enabling pre-computed context injection and hierarchical memory consolidation without re-ingestion. This collapses retrieval prefill from O(L) to O(1) and enables persistent, adjustable memory with O(1) session initialization.
This technique could significantly reduce latency and memory footprint for on-device or edge AI assistants, making persistent, context-aware language models more feasible on resource-constrained hardware. It may shift the competitive landscape for edge LLMs away from Transformer-based architectures.
Enables low-latency, memory-efficient, and persistent AI assistants on edge devices, reducing cloud dependency and improving privacy. Could lower the cost and complexity of deploying RAG-like capabilities in consumer electronics, wearables, and IoT.
Next signals to watch include: open-source implementations or demos of PRECOG/SMC on popular SSM backbones (e.g., Mamba), benchmarks comparing retrieval latency and memory usage against Transformer RAG, and adoption by edge AI chip or device manufacturers. Potential integration into on-device assistants from Apple, Google, or Samsung.