Credibility Assessment Based Byzantine-Resilient Decentralized Learning
Jian Hou, Fangyuan Wang, Chunling Wei, Hongyun Huang, Yong Hu, Ning Gui
Abstract
Decentralized deep learning has made significant success since it avoids the single point of failure in centralized solutions. However, the system might deviate from the correct model due to Byzantine attacks. Existing Byzantine-resilient defense models are mainly of a one-step evaluation fashion, making them vulnerable to rigorous topology and sophisticated cyber-attacks due to lack of historical evaluations. This paper proposes a credibility assessment based parameter aggregation rule (CA-PAR) that evaluates each neighboring node by its long-term performance. For each node and its neighbors, two concepts, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">immediate reward</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">history information based credibility</i> are firstly proposed to describe the immediate reliability at current iteration and the comprehensive assessment of the reliability respectively. Thereafter, all the received parameters are aggregated in linear combination, in which the adjacent weight is determined by credibility value. Finally, the influences of suspicious nodes can gradually be reduced and eliminated. Experimental results in MNIST and CIFAR-10 datasets indicate the algorithm’s tolerance for five state-of-the-art attack methods against an arbitrary number of faulty nodes. Compared with the previous defense models, the proposed algorithm in this paper outperforms in topology constraints, training accuracy and computation cost.