LatentPress: Context Compression Beyond Text and Vision
LatentPress introduces continuous memory tokens for context compression, achieving 4-16x compression with a small adapter (4.2M-26.2M parameters, ~0.1% of decoder). On LongMemEval, it reaches 0.504 accuracy at 7.70x compression versus 0.490 for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at 4-8x compression, while 16x trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is 5-9x faster than raw context or cached OCR. Transfer settings include zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived data.
LatentPress proposes a new context compression method that stores conversational histories and long documents as continuous memory tokens, which a frozen decoder reads directly through its input-embedding interface without text reconstruction at inference. A small reader-matched writer compresses 4-16x while training only an adapter with 4.2M-26.2M parameters, about 0.1% of the decoder. On LongMemEval, LatentPress achieves 0.504 accuracy at 7.70x compression, slightly above uncompressed evidence (0.490), and significantly outperforms text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at 4-8x compression, though 16x compression trails raw. Writing takes 43ms per conversation, about an order of magnitude faster than text summarization or OCR reconstruction, and reading is 5-9x faster than raw context or cached OCR. The interface is validated under two transfer settings: zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived data.
LatentPress shifts context compression from discrete text or image representations to continuous memory tokens that are directly consumed by a frozen decoder's input embedding layer. This bypasses the need for text reconstruction at inference, reducing latency and potentially preserving richer semantic information. The writer is a small adapter (4.2M-26.2M parameters) trained to produce tokens that the decoder can interpret, enabling 4-16x compression. The reported accuracy on LongMemEval at 7.70x compression (0.504) slightly exceeds uncompressed evidence (0.490), suggesting that the compressed representation may filter noise or emphasize salient information. The speed advantages (43ms writing, 5-9x faster reading) indicate practical efficiency gains. Transfer results show zero-shot capability from UltraChat to LongMemEval, but the abstract does not provide quantitative transfer performance, so further validation is needed.
This research addresses the growing need for efficient context management in long-conversation AI systems, where token limits and inference costs are significant constraints. By compressing context into continuous tokens, LatentPress could reduce memory and compute requirements for LLM-based applications such as chatbots, document analysis, and agentic systems. The approach may enable longer context windows without proportional increases in latency or cost. The use of a frozen decoder with a small trainable adapter suggests low adaptation cost, making it potentially attractive for deployment. However, the method is still in early research stages, and its performance on diverse real-world tasks and robustness to domain shift remain to be demonstrated.
LatentPress offers potential cost savings and performance improvements for AI services that process long documents or conversations. By compressing context 4-16x with minimal accuracy loss (and sometimes improvement), it could reduce token usage and inference latency, lowering operational costs for providers. The small adapter size (4.2M-26.2M parameters) means low training and storage overhead. The speed advantages (43ms writing, 5-9x faster reading) could enhance user experience in real-time applications. However, the technology is not yet productized, and its commercial viability depends on further validation and integration with existing LLM stacks.
Observable next signals include: (1) publication of full transfer results and additional benchmarks to assess generalization; (2) open-sourcing of code or model weights to enable community validation; (3) follow-up work extending LatentPress to multimodal contexts or larger decoders; (4) industry adoption or pilot projects in long-context applications; (5) comparisons with other continuous token compression methods such as Gist tokens or ICAE. If the method proves robust, it could influence the design of memory modules in LLM systems and reduce reliance on text summarization or retrieval.