A research paper proposes CASE (Context-Aware Semantic Embeddings), a framework that uses a custom-trained Gemma 3-based Tabular Language Model to generate dynamically contextualized row embeddings by pre-filling the KV cache with a representative sample of rows. Experiments on CARTE, TextTab, and TabArena benchmarks show substantial performance improvements for tabular learners on semantically rich datasets, especially in low-data regimes.
The paper introduces CASE, which bridges the semantic understanding of large language models with the statistical capabilities of tabular learners. Unlike prior methods that embed rows in isolation, CASE contextualizes embeddings by anchoring them in dataset semantics via a pre-filled KV cache. This approach resolves semantic ambiguities and improves performance on benchmarks with rich textual features.
CASE leverages a custom-trained Gemma 3-based Tabular Language Model and a KV cache pre-filling strategy to create persistent semantic anchors from a representative row sample. This allows row embeddings to be dynamically contextualized, capturing domain-specific semantics and resolving ambiguities that arise when treating textual features as discrete symbols.
The approach addresses a common limitation in tabular machine learning where textual features are underutilized. By integrating LLM-based semantic understanding, CASE could enhance applications in domains with mixed data types, such as healthcare, finance, and e-commerce, where tabular data often contains meaningful text.
Improved tabular learning on semantically rich datasets can lead to better predictive models in industries reliant on structured data with text, potentially reducing the need for extensive feature engineering and enabling more accurate insights from limited data.
Observable next signals include adoption of CASE-like methods in tabular deep learning libraries, further research on scaling the KV cache pre-filling to larger datasets, and integration with automated machine learning (AutoML) systems. Performance on low-data regimes suggests potential for few-shot learning in tabular domains.