Algorithmic Splitting: A Method for Dataset Preparation
Khalid M. Kahloot, Péter Ekler
Abstract
Datasets that appear in publications are curated and split into training, testing, and validation sub-datasets by domain experts. Consequently, machine learning models typically perform well on such split-by-hand datasets, whereas preparing real-world datasets into curated splits, i.e., training, testing, and validation sub-datasets, require extensive effort. Usually, random repetitive splitting is carried out, practiced, and evaluated until a better score is reached on the evaluation metrics. In this paper, a novel algorithmic method is proposed for splitting datasets for machine learning models. Algorithmic Splitting utilizes deterministic dimension reduction and density-based clustering techniques for estimating the data distribution. Afterward, splitting the dataset is carried out based on the estimated data distribution. Our objective is to achieve evenly representative splits of a given dataset in a standard and algorithmic way that reduces the perplexity of random splitting using the thorough splitting method. Experiments demonstrate the potential of Algorithmic Splitting through qualitative and quantitative evaluation of MNIST, Fashion MNIST, CIFAR-10, SmallNORB, and Shapes3D.