DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Researchers have introduced DP-LAC, a novel method designed to enhance privacy and efficiency in the federated fine-tuning of large language models (LLMs). While federated learning allows collaborative training across edge devices without sharing raw user data, it remains vulnerable to information leakage through client-provided gradients. Traditional differentially private stochastic gradient descent (DP-SGD) mitigates this by clipping contributions and adding noise, but existing adaptive techniques often require complex hyperparameter tuning that can deplete the privacy budget. DP-LAC addresses these limitations by initially estimating an optimal clipping threshold using private histogram estimation. It then dynamically adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results demonstrate that DP-LAC significantly outperforms both state-of-the-art adaptive clipping methods and standard DP-SGD, achieving an average accuracy gain of 6.6%. This advancement offers a more robust and lightweight solution for maintaining data privacy in distributed AI systems, particularly for resource-constrained edge devices.
Wire timeline
DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Researchers have introduced DP-LAC, a novel method designed to enhance privacy and efficiency in the federated fine-tuning of large language models (LLMs). While federated learning allows collaborative training across edge devices without sharing raw user data, it remains vulnerable to information leakage through client-provided gradients. Traditional differentially private stochastic gradient descent (DP-SGD) mitigates this by clipping contributions and adding noise, but existing adaptive techniques often require complex hyperparameter tuning that can deplete the privacy budget. DP-LAC addresses these limitations by initially estimating an optimal clipping threshold using private histogram estimation. It then dynamically adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results demonstrate that DP-LAC significantly outperforms both state-of-the-art adaptive clipping methods and standard DP-SGD, achieving an average accuracy gain of 6.6%. This advancement offers a more robust and lightweight solution for maintaining data privacy in distributed AI systems, particularly for resource-constrained edge devices.
cs.AI updates on arXiv.org