Personalized and Memory-Efficient Federated Learning via Parameter Efficient Fine Tuning Techniques for Next Word Prediction

Federated learning (FL) is a privacy-preserving model of distributed model learning, which allows multiple clients to jointly learn a common model without sharing raw data. Nonetheless, it is still computationally intensive and memory-limited to be able to train the large-scale language models like GPT-2 over edge devices. To resolve this, we propose a personalized and memory-efficient federated learning model, which combines Parameter-Efficient Fine-Tuning (PEFT) methods- HeLoRA, BitFit, and Prompt Tuning to the next-word prediction tasks. On Shakespeare data, with the GPT-2 Medium model (345M parameters) , we model heterogeneous client environments, which are instances of different characters. The framework is a combination of FedPer-style personalization whereby global base layers are shared and personal layers are local and FedAvg PEFT parameter aggregation. Empirical evidence indicates that HeLORA has the most favorable trade off between accuracy, perplexity and the cost of communication, which is 99.39 percent in memory efficiency with far less resource consumption than full fine-tuning. The research confirms that PEFT-based federated personalization is a scalable and privacy-aware method to use next-word prediction with respect to distributed settings.