FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation

Read full text Summarise with VIRAT
Authors: Fatema Siddika, Md. Anwar Hossen, J. Pablo Muñoz, Tanya G. Roosta, Anuj Kumar Sharma, Ali Jannesari
Cited by: 0
References: 53

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Parameter-efficient fine-tuning (PEFT) adapts large pre-trained models by updating only a small subset of parameters.Recently, Representation Fine-Tuning (ReFT) has emerged as an effective alternative.ReFT shifts the fine-tuning paradigm from updating model weights to directly manipulating hidden representations that capture rich semantic information, and outperform state-of-the-art PEFTs in standalone settings.However, its application in Federated Learning (FL) remains challenging due to heterogeneity in clients' data distributions, model capacities, and computational resources.To address these challenges, we introduce Federated Representation Fine-Tuning (FedReFT), a novel approach to fine-tune clients' hidden representations.Fe-dReFT applies sparse intervention layers to steer hidden representations directly, offering a lightweight and semantically rich fine-tuning alternative ideal for edge devices.However, representation-level updates are especially vulnerable to aggregation mismatch under different task heterogeneity, where naive averaging can corrupt semantic alignment.To mitigate this issue, we propose All-But-Me (ABM) aggregation, where each client receives the aggregated updates of others and partially incorporates them, enabling stable and personalized learning by balancing local focus with global knowledge.We further design an adaptive update strategy inspired by Test-Time Computing (TTC) to balance local and global contributions under heterogeneous conditions.Fe-dReFT achieves state-of-the-art performance on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks, while delivering 1×-49× higher parameter efficiency compared to leading LoRA-based methods.The paper code is available at Anonymous Repository

Fields of Study & Subjects

Privacy-Preserving Technologies in Data Domain Adaptation and Few-Shot Learning Topic Modeling

Publication details

Journal / source
Findings of the Association for Computational Linguistics: EACL
Pages
4341–4362
Publication date
1 January 2026
Language
EN