Model Stability Under Client Diversity: A Comparative Study of Federated Aggregation Algorithms in Bangla NLP

Federated Learning (FL) has received a lot of attention lately due to its capacity to protect data privacy. FL is a distributed learning technique in which several clients work together under the supervision of a central server to train a common global model without ever disclosing their raw data. Rather, the server securely aggregates locally computed model updates, like gradients or parameters, that are shared by each client. This configuration allows decentralised, privacy-aware knowledge sharing across a range of devices and institutions while also safeguarding data privacy and integrity. Despite its advantages, FL faces several challenges, with client heterogeneity being one of the most critical. In real-world applications, clients often have datasets that differ in size, distribution, and balance—commonly referred to as IID (Independent and Identically Distributed) and non-IID scenarios. Such differences can lead to client drift and negatively impact model convergence. In this study, we explore how different federated weight aggregation algorithms perform under varying levels of client heterogeneity. Specifically, we compare FedAvg, FedProx, and Fed-Nova on both IID and non-IID data partitions. Our experimental results demonstrate that FedProx consistently outperforms the other methods in both settings, for its proximal regularization term, which stabilizes updates and reduces divergence among heterogeneous clients.