Scalable Machine Learning Infrastructure on Cloud for Large-Scale Data Processing
Jigar Shah
Abstract
This research focuses on exploring the state of affairs using advanced computing paradigms that include cloud computing scenario, quantum computing, and HPC for vast machine learning and remote sensing solutions. Aspects of this conventional cloud-based machine learning model’s limitations are discussed, with an introduction to collaborative machine learning frameworks, as well as how they operate with on-device resources and cloud environments. In this article, to provide a convenient API to the application insertion layer, deployment, data pipelines, and optimized compute containers such as Walle are discussed as end-to-end systems. It also examines the applicability of HPC, cloud, and quantum computing resources to adequately manage vast RS datasets, train complex DL models, and facilitate crucial practical applications across various industries and research areas, including environmental management and sustainable urban development. New trends are highlighted; opportunities also consist of a possibility to incorporate the edge computing concept and the fact that further advancements require collaboration across multiple disciplines. DOI: https://doi.org/10.52783/tjjpt.v42.i2.7166