Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
A research paper introduces SQD (SubQuadratic Disaggregation), a fine-grained heterogeneous disaggregation scheme that splits decode by quadratic and subquadratic attention rather than by operator type. It applies across subquadratic attention variants. For sparse attention LLMs, decode is disaggregated into top-k selection and top-k attention plus FFN. For linear and sliding-window attention LLMs, decode is disaggregated into dense attention layers and subquadratic attention layers plus FFN. In an adjusted 8xB200 heterogeneous system proxy, average tokens/J improvements of 53% are observed on GLM 5.2.
Frontier language models are increasingly using subquadratic attention to reduce memory footprint and compute requirements during inference while maintaining frontier accuracy. Existing systems make dense attention-centric disaggregated serving decisions. The paper shows that disaggregating inference around the unique arithmetic intensity and memory footprint of subquadratic attention LLMs can achieve significant throughput and energy efficiency gains on emerging DRAM-based and SRAM-only heterogeneous systems. The proposed SQD scheme splits decode by quadratic and subquadratic attention, with specific strategies for sparse, linear, and sliding-window attention variants. In an adjusted 8xB200 heterogeneous system proxy, average tokens/J improvements of 53% are observed on GLM 5.2.
SQD disaggregates decode based on attention type rather than operator type, enabling fine-grained placement of compute and memory across heterogeneous systems. For sparse attention, top-k selection requires indexing through the full KV cache, while top-k attention and FFN have static memory footprints, allowing separate scheduling. For linear and sliding-window attention, dense attention layers are separated from subquadratic attention layers plus FFN. This approach leverages the distinct arithmetic intensity and memory access patterns of subquadratic attention to improve energy efficiency.
The shift toward subquadratic attention in frontier models is driving a need for new serving infrastructure. Existing disaggregated serving systems are optimized for dense attention and may not fully exploit the efficiency opportunities of subquadratic attention. SQD demonstrates that rethinking disaggregation around attention type can yield significant energy efficiency gains, which is critical as inference costs and energy consumption become major concerns for large-scale deployment.
The 53% improvement in tokens per joule on an 8xB200 system proxy suggests substantial energy cost savings for inference providers. This could reduce operational expenses and improve the sustainability of AI services. The approach may also enable higher throughput on existing hardware, potentially delaying or reducing the need for hardware upgrades.
As subquadratic attention becomes more prevalent, serving systems will need to adapt. The SQD approach may influence the design of future inference serving frameworks and hardware-software co-design. Further research may explore additional subquadratic attention variants and heterogeneous system configurations. Adoption of such schemes could lead to more energy-efficient and cost-effective deployment of large language models.