Design Scalable Data Pipelines For Ai Applications
Yeshwanth Vasa, Santosh Jaini, Prudhvi Singirikonda
Abstract
The specific focus of this paper is to review the data pipeline at a larger scale for the use of AI, which addresses the '3V's, that is, volume, variety, and velocity. This research aims to discover and compare architectural approaches and tools that enhance the speed, solidity, and scalability of data feeds for AI. These include working in hardware accelerators, cloud-based working, and the approaches to managing data right from ingestion to production. It is essential to demonstrate that the requirements for large-scale solutions such as FPGAs, Containers, and model life cycle management can improve the latency and throughput of an AI data stream using simulations and real-time scenarios. The study shows the importance of combining these advanced technologies to cover the standard challenges in AI data analysis, including the difficulties in data processing and the necessity of real-time analysis. Lastly, the study identifies optimal practices for implementing data pipelines when choosing the most accurate and effective way to feed the growing demand for artificial intelligence models while ensuring the support needed to run and maintain those AI-driven models in production. Published: 26 January 2021