Scaling graph traversal to 281 trillion edges with 40 million cores
Huanqi Cao, Yuanwei Wang, Haojie Wang, Heng Lin, Zixuan Ma, Wanwang Yin, Wenguang Chen
Abstract
Graph processing, especially high-performance graph traversal, plays a more and more important role in data analytics. The successor of Sunway TaihuLight, New Sunway, is equipped with nearly 10 PB memory and over 40 million cores, which brings the opportunity to process hundreds of trillions of edges graphs. However, the graph with an unprecedented scale also brings severe performance challenges, including load imbalance, poor locality, and irregular access of graph traversal workload.
Topics & Concepts
Graph traversalTree traversalComputer scienceLocalityParallel computingGraphAnalyticsWorkloadScalingTheoretical computer scienceOperating systemData scienceMathematicsAlgorithmPhilosophyGeometryLinguisticsGraph Theory and AlgorithmsAdvanced Graph Neural NetworksCloud Computing and Resource Management