Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models
Juncai Liu, Jessie Hui Wang, Yimin Jiang
Abstract
Scaling models to large sizes to improve performance has led a trend in deep learning, and sparsely activated Mixture-of-Expert (MoE) is a promising architecture to scale models. However, training MoE models in existing systems is expensive, mainly due to the All-to-All communication between layers.
Topics & Concepts
Computer scienceArtificial intelligenceJanusTraining (meteorology)Scale (ratio)ScalingMachine learningDeep learningArchitectureDistributed computingGeometryProgramming languageVisual artsArtQuantum mechanicsMathematicsMeteorologyPhysicsMobile Crowdsensing and CrowdsourcingAir Quality Monitoring and ForecastingData Stream Mining Techniques