On the Existence of Global Minima and Convergence Analyses for Gradient Descent Methods in the Training of Deep Neural Networks

Abstract

Although gradient descent (GD) optimization methods in combination with rectified linear unit (ReLU) artificial neural networks (ANNs) often supply an impressive performance in real world learning problems, till this day it remains -in all practically relevant scenarios -an open problem of research to rigorously prove (or disprove) the conjecture that such GD optimization methods do converge in the training of ANNs with ReLU activation.

Topics & Concepts