Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers
Yangyang Feng, Minhui Xie, Zijie Tian, Shuo Wang, Youyou Lu, Jiwu Shu
Abstract
Fine-tuning on cheap commodity GPU servers makes large-scale deep learning models benefit more people. However, the low inter-GPU communication bandwidth and pressing communication contention on the commodity GPU server obstruct training efficiency.
Topics & Concepts
ServerComputer scienceCommodityScale (ratio)Bandwidth (computing)General-purpose computing on graphics processing unitsDeep learningParallel computingArtificial intelligenceComputer networkOperating systemGraphicsQuantum mechanicsEconomicsMarket economyPhysicsAdvanced Neural Network ApplicationsStochastic Gradient Optimization TechniquesFerroelectric and Negative Capacitance Devices