Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
The paper 'Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving' proposes SALT, a hierarchical fine-tuning framework for multi-tenant LoRA serving. It addresses the trade-off between serving efficiency and task performance by using high-capacity domain centroids trained with an alignment regularizer, and ultra-low-rank task residual adapters. During inference, centroids are pinned in GPU VRAM while residuals are dynamically swapped, reducing VRAM footprint and PCIe overhead while maintaining performance.
Researchers introduce Subspace-Aligned LoRA Training (SALT), a three-phase framework for efficient multi-tenant LoRA serving. A provider trains domain centroids on public data with a subspace alignment regularizer; users fine-tune ultra-low-rank residual adapters on private data atop frozen centroids; at inference, centroids stay pinned in GPU VRAM and only task residuals are swapped. This approach minimizes VRAM and PCIe transfer costs while preserving downstream task performance across various LLM scales.
SALT leverages a novel alignment regularizer to cohere in-domain task subspaces into a unified basis, enabling effective decomposition into shared centroids and ultra-low-rank residuals. This allows high-rank representational capacity to be retained in the pinned centroid while keeping per-user adapter rank extremely low (r ≤ 2), drastically reducing per-adapter memory and swap overhead without sacrificing task performance.
This method directly addresses the scalability bottleneck in serving numerous fine-tuned LLM variants. By decoupling shared domain knowledge from user-specific adaptations, it enables cost-effective, high-throughput multi-tenant serving, which is critical for LLM-as-a-service platforms and enterprise deployments where hundreds of customized adapters must be served concurrently.
For AI service providers, SALT can significantly reduce GPU VRAM requirements and PCIe bandwidth usage, lowering infrastructure costs while maintaining high-quality personalized model serving. This enables more tenants per GPU, improving margins and scalability for fine-tuning-as-a-service offerings.
Observable next signals include adoption by LLM serving frameworks (e.g., vLLM, TGI), integration with model hubs for standardized centroid sharing, and extensions to other parameter-efficient fine-tuning methods. Further research may explore dynamic centroid selection and privacy-preserving centroid training across multiple data providers.