Building Trust Beyond Update Divergence: Dual-Refined Aggregation for Byzantine-Robust Federated Learning

Federated learning (FL) enables collaborative training across distributed clients but remains vulnerable to Byzantine attacks, especially stealthy ones. The threat is even amplified in non-IID settings, where client heterogeneity causes greater divergence in feature distributions and inter-client distances. Existing defenses often rely on strong assumptions or raw update distances, limiting their effectiveness under such heterogeneity. To address this gap, we proposeFedRefiner, a decoupled dual-refined aggregation algorithm designed to mitigate stealthy attacks on heterogeneous data. Our intuition is that the significance distribution of client updates reveals subtle malicious evasion, altering critical features for attack while perturbing unimportant ones, thereby exposing true inter-client distances.FedRefinergoes beyond norm-based filtering by refining both weighted scores and aggregated updates, enabling more accurate distinction between malicious behavior and benign non-IID variation. It first derives significance distribution vectors as refined updates by sparsity, then clusters them to compute weighted similarity scores for group reliability. These clusters then align raw updates into groups for group-wise refinement, yielding robust aggregated updates. We theoretically prove the convergence ofFedRefinerunder Byzantine attacks in non-IID settings. Extensive evaluation on 8 datasets against 10 attacks (including 2 adaptive ones) and 13 defenses shows thatFedRefineroutperforms state-of-the-art defenses, achieving up to a 10% gain in overall accuracy and a 14.8% improvement in worst-case performance under both IID and non-IID settings. Ablation studies further demonstrate its robustness across different hyperparameters, attacker ratios, data heterogeneity, and model/client scales, while incurring low computation and no storage overhead.