Tackling Long-Tailed Distribution Issue in Graph Neural Networks via Normalization
Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, Qi Yuan, Jieping Ye
Abstract
Graph Neural Networks (GNNs) have attracted much attention due to their superior learning capability. Despite the successful applications of GNNs in many areas, their performance suffers heavily from the long-tailed node degree distribution. Most prior studies tackle this issue by devising sophisticated model architectures. In this article, we aim to improve the performance of tail nodes (low-degree or hard-to-classify nodes) via a generic and light normalization method. In detail, we propose a novel normalization method for GNNs, termed as ResNorm, which <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Res</b> hapes a long-tailed distribution into a normal-like distribution via <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Norm</b> alization. The ResNorm includes two operators. First, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scale</i> operator reshapes the distribution of the node-wise standard deviation (NStd) so as to improve the accuracy of tail nodes. Second, the analysis of the behavior of the standard shift indicates that the standard shift serves as a preconditioner on the weight matrix, increasing the risk of over-smoothing. To address this issue, we design a new <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">shift</i> operator for ResNorm, which simulates the degree-specific parameter strategy in a low-cost manner. Extensive experiments on various node classification benchmark datasets have validated the effectiveness of ResNorm in improving the performance of tail nodes as well as the overall performance.