LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications
Jinhan Xin, Kai Hwang, Zhibin Yu
2022Proceedings of the 2022 International Conference on Management of Data29 citationsDOIOpen Access PDF
Abstract
Spark SQL has been widely deployed in industry but it is challenging to tune its performance. Recent studies try to employ machine learning (ML) to solve this problem, but suffer from two drawbacks. First, it takes a long time (high overhead) to collect training samples. Second, the optimal configuration for one input data size of the same application might not be optimal for others.
Topics & Concepts
SPARK (programming language)Computer scienceSQLBenchmark (surveying)Overhead (engineering)Node (physics)Operating systemDatabaseEngineeringProgramming languageGeodesyGeographyStructural engineeringCloud Computing and Resource ManagementScientific Computing and Data ManagementMachine Learning and Data Classification