Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architectures

The rise of foundation models (FMs) has reshaped the landscape of machine learning. As these models continue to grow, leveraging geo-distributed data from wireless devices has become increasingly critical, giving rise to federated foundation models (FedFMs). More recently, FMs have evolved into multi-modal multi-task (M3T) FMs (e.g., GPT-4) capable of processing diverse modalities across multiple tasks, which motivates a new underexplored paradigm: M3T-FedFMs. In this article, we unveil an unexplored variation of M3T-FedFMs by proposing hierarchical federated foundation models (H-Fed- FMs), which in turn expose two overlooked heterogeneity dimensions to fog/edge networks that have a direct impact on these emerging models: (i) heterogeneity in collected modalities and (ii) heterogeneity in executed tasks across fog/edge nodes. H-FedFMs strategically align the modular structure of M3T-FMs, comprising modality encoders, prompts, mixture-of-experts (MoEs), adapters, and task heads, with the hierarchical nature of fog/edge infrastructures. Moreover, H-FedFMs enable the optional usage of device-to-device (D2D) communications, enabling horizontal module relaying and localized cooperative training among nodes when feasible. Through delving into the architectural design of H-FedFMs, we highlight their unique capabilities along with a series of tailored future research directions. Additionally, to demonstrate their potential, we prototype H-FedFMs in a wireless network setting and show that they achieve notable cost savings (i.e., energy and latency) compared to existing baselines. We also release the open-source implementation of H-FedFMs to foster further exploration in this emerging area (GitHub: https://github.com/ payamsiabd/M3T-FFM).