HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
HCRMap is a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. It dynamically determines expert promotion, retention, demotion, or eviction based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure. Experimental results show HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in prefill and decode stages.
HCRMap proposes a pressure-aware hot-expert residency mapping framework for 3.5D MoE chiplet inference, dynamically managing expert replicas across memory tiers to mitigate communication, memory, and queue bottlenecks, achieving 43.6% and 43.0% latency reduction over Hydra.
HCRMap addresses expert hotness skew in MoE inference by jointly considering compute imbalance and pressure on communication, memory, I/O, and execution queues. The framework's dynamic replica management across memory tiers is a novel approach. Next signal: validation on larger chiplet configurations or real hardware.
This work targets the growing need for efficient inference of large MoE models on multi-chiplet systems, a key infrastructure challenge for AI deployment. Next signal: adoption by chiplet hardware vendors or integration into inference serving systems.
Reducing inference latency by over 40% directly lowers operational costs and improves user experience for MoE-based services. Next signal: licensing or open-source release of HCRMap implementation.
HCRMap could influence future chiplet interconnect designs and memory hierarchy optimizations for MoE models. Next signal: follow-up work extending to other model architectures or heterogeneous memory systems.