Variational Reinforcement Learning for Hyper-Parameter Tuning of Adaptive Evolutionary Algorithm
Haotian Zhang, Jianyong Sun, Yuhao Wang, Jialong Shi, Zongben Xu
Abstract
The performance of an evolutionary algorithm (EA) is deeply affected by its control parameter's setting. It has become a trend in recent studies to treat the control parameter as a random variable. In these studies, the associated distribution of the control parameter is updated at each generation and new parameter setting is sampled from the distribution. The distribution's parameter (called hyper-parameter) is thus critical to the algorithmic performance. In this paper, we propose a variational learning framework to tune the hyper-parameters of EA, in which the expectation-maximization (EM) algorithm and a reinforcement learning algorithm are combined. To verify the effectiveness of the proposed method which is named Reinforcement EM (REM), we apply it to tune the hyper-parameters of the distributions of two important parameters, i.e. the scaling parameter ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F$</tex-math></inline-formula> ) and crossover rate ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$CR$</tex-math></inline-formula> ), of differential evolution (DE) and an adaptive DE algorithm. In addition, we propose to use the meta-learning technique to learn good initial distributions for the hyper-parameters of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$CR$</tex-math></inline-formula> so that the REM can effectively adapt to a new optimization problem. Experimental results obtained on the CEC 2018 test suite show that with the tuned hyper-parameters, DE and the adaptive DE can achieve significantly better performance than their counterparts with empirical parameter settings and with parameters tuned by some widely-used tuning methods, including ParamILS, F-Race and Bayesian optimization algorithm.