Explainable Benchmarking for Iterative Optimization Heuristics
Bas van Stein, Diederick Vermetten, Anna V. Kononova, Thomas Bäck
Abstract
Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This article presents a novel approach that we call explainable benchmarking. We introduce the IOHxplainer software library, for systematic analysing the performance of various optimization algorithms and the impact of their different components and hyperparameters. We showcase the methodology in the context of two modular optimization implementations. Through this library, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, aiming to improve future benchmarking and algorithm design practices.