Hercules: Boosting the Performance of Privacy-Preserving Federated Learning
Guowen Xu, Xingshuo Han, Shengmin Xu, Tianwei Zhang, Hongwei Li, Xinyi Huang, Robert H. Deng
Abstract
In this paper, we address the problem of privacy-preserving federated neural network training with <inline-formula><tex-math notation="LaTeX">$N$</tex-math></inline-formula> users. We present <b>Hercules</b> , an efficient and high-precision training framework that can tolerate collusion of up to <inline-formula><tex-math notation="LaTeX">$N-1$</tex-math></inline-formula> users. <b>Hercules</b> follows the POSEIDON framework proposed by Sav et al. (NDSS’21), but makes a qualitative leap in performance with the following contributions: (i) we design a novel parallel homomorphic computation method for matrix operations, which enables fast Single Instruction and Multiple Data (SIMD) operations over ciphertexts. For the multiplication of two <inline-formula><tex-math notation="LaTeX">$h\times h$</tex-math></inline-formula> dimensional matrices, our method reduces the computation complexity from <inline-formula><tex-math notation="LaTeX">$O(h^{3})$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$O(h)$</tex-math></inline-formula> . This greatly improves the training efficiency of the neural network since the ciphertext computation is dominated by the convolution operations; (ii) we present an efficient approximation on the sign function based on the composite polynomial approximation. It is used to approximate non-polynomial functions (i.e., <monospace>ReLU</monospace> and <monospace>max</monospace> ), with the optimal asymptotic complexity. Extensive experiments on various benchmark datasets (BCW, ESR, CREDIT, MNIST, SVHN, CIFAR-10 and CIFAR-100) show that compared with POSEIDON, <b>Hercules</b> obtains up to 4% increase in model accuracy, and up to <inline-formula><tex-math notation="LaTeX">$60\times$</tex-math></inline-formula> reduction in the computation and communication cost.