ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
ACE is a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. It includes Global Spectral Proxy (GSP) and Router-Conditioned Refinement (RCR). During inference, ACE skips an expert slot only when both views identify it as low-contribution, while always retaining the top-1 expert.
Mixture-of-Experts (MoE) architectures scale LLMs efficiently, but fixed top-k routing activates the same number of expert slots per token, causing redundant computation. Existing expert-skipping methods rely on router confidence, calibration data, or additional training, and cannot reliably estimate actual expert contribution. ACE addresses this with two components: GSP estimates global transformation capacity from coupled gate, up, and down projections with RMSNorm scaling; RCR constructs expert-specific direction prototypes from centered router weights and evaluates expert responses along routing-preferred directions. ACE combines both estimates with runtime router gates and skips an expert slot only when both views identify it as low-contribution, always retaining the top-1 expert.
ACE introduces a dual-view contribution estimation: GSP captures global transformation capacity via spectral analysis of coupled projections, while RCR uses router weight geometry to assess expert alignment. The requirement that both views agree before skipping provides a conservative gating mechanism that may reduce false negatives in expert pruning. The method is checkpoint-preserving, meaning it can be applied to existing MoE models without retraining or calibration.
This research targets inference efficiency for MoE-based LLMs, a key cost driver in deployment. A calibration-free, training-free method lowers adoption barriers for model providers seeking to reduce serving costs without modifying model weights. If validated, it could influence inference optimization practices across cloud and edge deployments.
Reducing redundant expert computation can lower inference latency and cost for MoE LLMs, improving margins for API providers and enabling more efficient on-device deployment. The checkpoint-preserving nature avoids expensive retraining, offering immediate cost savings if the method proves robust.
Next signals include empirical results on benchmark accuracy and speedup across different MoE models, comparisons with existing expert-skipping baselines, and potential integration into inference frameworks. Adoption by model serving platforms or open-source libraries would indicate practical viability.