Dynamical Systems–Based Neural Networks
Elena Celledoni, Davide Murari, Brynjulf Owren, Carola‐Bibiane Schönlieb, Ferdia Sherry
Abstract
.Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a nonautonomous ODE and build neural networks using a suitable, structure-preserving, numerical time discretization. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modeling procedure allows a better theoretical understanding of their behavior. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.Reproducibility of computational results.This paper has been awarded the "SIAM Reproducibility Badge: code and data available", as a recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available in https://github.com/davidemurari/StructuredNeuralNetworks OR in the Supplementary Materials (123366_2_supp_531986_rxfx2s_sc.pdf [736KB]).Keywordsneural networksdynamical systemsLipschitz networksstructure-preserving deep learninguniversal approximation theoremMSC codes65L0565L0637M15