Recurrent Neural Networks Are Universal Approximators With Stochastic Inputs

Xiuqiong Chen, Yangtianze Tao, Wenjie Xu, Stephen Shing-Toung Yau

2022IEEE Transactions on Neural Networks and Learning Systems20 citationsDOI Open Access PDF

Abstract

In this article, we investigate the approximation ability of recurrent neural networks (RNNs) with stochastic inputs in state space model form. More explicitly, we prove that open dynamical systems with stochastic inputs can be well-approximated by a special class of RNNs under some natural assumptions, and the asymptotic approximation error has also been delicately analyzed as time goes to infinity. In addition, as an important application of this result, we construct an RNN-based filter and prove that it can well-approximate finite dimensional filters which include Kalman filter (KF) and Beneš filter as special cases. The efficiency of RNN-based filter has also been verified by two numerical experiments compared with optimal KF.

Topics & Concepts

Recurrent neural networkKalman filterFilter (signal processing)Control theory (sociology)Computer scienceState spaceMathematicsConstruct (python library)Artificial neural networkRecursive filterState (computer science)Extended Kalman filterClass (philosophy)Filter designSpace (punctuation)Dynamical systems theoryApproximation errorDiscrete time and continuous timeAlgorithmStochastic processDynamical system (definition)Applied mathematicsFiltering theoryMathematical optimizationState-space representationFiltering problemStochastic approximationTrajectoryNeural Networks and ApplicationsNeural Networks Stability and SynchronizationMachine Learning and ELM