An uncertainty-based objective function for hyperparameter optimization in Gaussian processes applied to expensive black-box problems
Pietro Lualdi, Ralf Sturm, Andrés Camero, Tjark Siefkes
Abstract
As of today, Gaussian processes (GPs) have been widely and successfully used in the context of design optimization based on expensive-to-evaluate functions. This supervised learning method enables a generation of accurate nonlinear surrogate models based on relatively small datasets. Nonetheless, their most valuable asset is to provide uncertainty in predictions. Despite their excellent stochastic properties, Gaussian processes are unfortunately not immune to threats such as the generation of distorted predictions, especially when the amount of data available is very limited. This shortcoming is caused by a poor choice of the GP hyperparameters and represents a serious threat to the efficiency and effectiveness of the whole surrogate-based optimization. In this paper we present the Hybrid Loss (HL), a novel uncertainty-aware objective function for the hyperparameter tuning of Gaussian processes. This method is intended to exploit information coming from the predictive variance to remedy the typical shortcomings of the log marginal likelihood, i.e. the objective function commonly used to optimize GP hyperparameters. By pairing this methodology with a well-known adaptive sampling strategy, we investigate the performance on a wide range of benchmark functions and a real engineering problem. The observed evidence clearly shows how uncertainty can be successfully exploited to make a wiser choice of the hyperparameters. This translates into more accurate predictions, surrogate models less prone to overfitting, and above all, greatly improved convergence rates.