DP-Prox: A Robust and Differentially Private Framework for Federated Instruction Tuning of Small LLMs on 8 GB Edge Devices

This paper introduces DP-Prox, an end-to-end framework for robust and differentially private federated instruction tuning of small Large Language Models (LLMs) on 8 GB edge devices. DP-Prox is instantiated using the 3.8 billionparameter Phi-3-mini model with 4-bit Quantized Low-Rank Adaptation (QLoRA), and combines (i) FedProx-style proximal regularization on low-rank adapters to stabilize optimization under highly Non-IID smart-home data, and (ii) Differentially Private SGD (DP-SGD) applied directly to adapter gradients to provide user-level$(\epsilon, \delta)$-differential privacy against an honest-but-curious server. In a realistic IoT instruction-following benchmark with Dirichlet-skewed clients ($\alpha=0.3$), DP-Prox improves F1-score from 0.80 (Edge-FIT with FedAvg) to 0.82 while reducing convergence rounds from$T=50$to$T \approx 40$. Under a strict privacy budget$\left(\epsilon=5, \delta=10^{-5}\right)$, DP-Prox maintains an F1-score of 0.77 with only$+15 \%-20 \%$local overhead, and preserves the communication efficiency of QLoRA-based federated training. Qualitative analyses further show that DP-Prox yields more consistent instruction-following behavior across heterogeneous homes, especially on multi-device routines and privacy-sensitive queries. Together, these results position DP-Prox as a practical and regulation-ready blueprint for deploying small, edge-resident LLMs in smart-home IoT environments.