Varuna
Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra
Abstract
Systems for training massive deep learning models (billions of parameters) today assume and require specialized "hyperclusters": hundreds or thousands of GPUs wired with specialized high-bandwidth interconnects such as NV-Link and Infiniband. Besides being expensive, such dependence on hyperclusters and custom high-speed inter-connects limits the size of such clusters, creating (a) scalability limits on job parallelism; (b) resource fragmentation across hyperclusters.
Topics & Concepts
InfiniBandComputer scienceScalabilityBandwidth (computing)Distributed computingComputer architectureFragmentation (computing)Parallel computingComputer networkOperating systemParallel Computing and Optimization TechniquesFerroelectric and Negative Capacitance DevicesCloud Computing and Resource Management