Sanitizing Sentence Embeddings (and Labels) for Local Differential Privacy
Minxin Du, Xiang Yue, Sherman S. M. Chow, Huan Sun
Abstract
Differentially private (DP) learning, notably DP stochastic gradient descent (DP-SGD), has limited applicability in fine-tuning gigantic pre-trained language models (LMs) for natural language processing tasks. The culprit is the perturbation of gradients (as gigantic as entire models), leading to significant efficiency and accuracy drops.
Topics & Concepts
Computer scienceDifferential privacySentenceStochastic gradient descentArtificial intelligenceLanguage modelGradient descentNatural language processingSpeech recognitionAlgorithmArtificial neural networkPrivacy-Preserving Technologies in DataCryptography and Data SecurityArtificial Intelligence in Healthcare and Education