Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
A research paper on arXiv demonstrates that multiple Intel AI PCs can serve large language models beyond the memory capacity of a single machine by using pipeline parallelism with pre-compiled OpenVINO shards. The paper reports that a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput.
The paper proposes a method to run large language models on fleets of Intel AI PCs by splitting the model into per-stage shards, each pre-compiled into an OpenVINO graph. Three techniques are used to improve performance: injecting a beam_idx Gather to trigger IndirectKVCache fusion, leveraging speculative decoding on stateful OpenVINO models, and micro-batching to interleave requests across stages. The approach enables serving models like Llama 3.1 8B INT4 across two nodes with improved throughput for concurrent users.
The key technical contribution is recovering monolithic inference speed in sharded OpenVINO models by injecting a beam_idx Gather into each shard, which triggers the IndirectKVCache fusion optimization. This addresses a performance gap in naive per-stage exports. Additionally, speculative decoding and micro-batching are employed to improve throughput in a distributed pipeline setting.
This research suggests that idle consumer AI PCs with integrated GPUs and NPUs could be pooled to serve large models, potentially reducing the need for dedicated high-memory inference hardware. It may enable edge or distributed inference scenarios where a group of standard PCs collaborates over a local network.
The approach could lower infrastructure costs for serving large language models by utilizing existing AI PC fleets, especially in environments with idle compute resources. It may enable new deployment models for enterprises or edge computing providers.
Observable next signals include adoption of this technique in OpenVINO tooling, benchmarks on larger models or more nodes, and commercial products that leverage fleets of AI PCs for inference. Further research may explore fault tolerance, load balancing, and support for heterogeneous hardware.