Optimizing Large-Scale ML Training using Cloud-based Distributed Computing
Yasodhara Varma, Manivannan Kothandaraman
Abstract
Large-scale machine learning (ML) models development depend on a strong and effective cloud- based infrastructure. This work investigates best ways to create cloud-native machine learning training environments using Azure, GCP, and AWS. We underline how important distributed computing systems as Apache Spark, Dask, and Ray are in handling large amounts of data and accelerating training times. Apart from performance, one major issue is cost control. We investigate using spot instances, auto-scaling & the storage improves the equilibrium between computing capability & budgetary constraints. We present a useful case study on Dask on AWS EMR financial fraud detection. This case study shows how distributed machine learning workloads may be optimized for maximum efficiency, therefore lowering time and cost while maintaining model correctness by means of this reduction of the burden. This article presents doable solutions for cloud architects, data scientists, and ML engineers on creating a scalable and reasonably affordable ML training pipeline in the cloud