FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
FraQ is a method for federated fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA). It addresses the aggregation mismatch that occurs when naively averaging LoRA factors across clients. FraQ performs exact aggregation in the full weight space and then recompresses the result efficiently in coordinate space. It factorizes the aggregate into an orthonormal basis and a compact coordinate matrix, recovers the singular spectrum from a small Gram matrix, selects the smallest rank satisfying an energy threshold, and maps the selected coordinate subspace back through the basis to construct the global adapter. Experiments on text classification and commonsense reasoning benchmarks show accuracy close to uncompressed baselines while substantially reducing downlink communication.
FraQ introduces an efficient coordinate-space recompression technique for federated LoRA, enabling accurate global model aggregation without the computational and memory costs of full decomposition. It achieves near-baseline accuracy on text classification and commonsense reasoning tasks while significantly cutting downlink communication.
FraQ avoids the aggregation mismatch of naive LoRA averaging by forming the exact aggregate in weight space and then recompressing via coordinate-space factorization. It uses an orthonormal basis and a compact coordinate matrix, recovers singular values from a Gram matrix, and selects rank based on an energy threshold, balancing accuracy and compression.
This method could lower communication costs in federated learning deployments, making it more practical to fine-tune LLMs across distributed, privacy-sensitive data sources without centralizing data.
FraQ reduces downlink communication in federated LoRA, which can lower bandwidth costs and enable more efficient collaborative model improvement across organizations handling sensitive data.
Next signals include potential adoption in privacy-focused enterprise AI solutions, integration into federated learning frameworks, and further validation on larger models and diverse tasks.