Morphling
Luping Wang, Lingyun Yang, Yinghao Yu, Wei Wang, Bo Li, Xianchao Sun, Jian He, Liping Zhang
Abstract
Machine learning models are widely deployed in production cloud to provide online inference services. Efficiently deploying inference services requires careful tuning of hardware and runtime configurations (e.g., GPU type, GPU memory, batch size), which can significantly improve the model serving performance and reduce cost. However, existing autoconfiguration approaches for general workloads, such as Bayesian optimization and white-box prediction, are inefficient in navigating the high-dimensional configuration space of model serving, incurring high sampling cost.
Topics & Concepts
Computer scienceInferenceCloud computingBayesian optimizationBayesian inferenceDistributed computingBayesian probabilityMachine learningComputer engineeringComputer architectureArtificial intelligenceOperating systemParallel Computing and Optimization TechniquesMachine Learning and Data ClassificationAdvanced Neural Network Applications