Efficient Training of Feed-Forward Neural Networks
Martin Möller
Abstract
Learning in neural networks can be formulated in terms of the minimization of an error function E. The problem of minimizing multivariate, continuous, differentiable functions is one which has been widely studied, and some of the conventional approaches to this problem are directly or with small modifications applicable to the training of neural networks. However, training neural networks differs from conventional optimization problems on two important points: The number of variables to be optimized is usually several magnitudes larger than the number of variables the conventional optimization algorithms were designed to optimize. The derivative information needed to perform the optimization in each iterative step is computationally heavy and is calculated based on very special purpose algorithms.