On Transversality of Bent Hyperplane Arrangements and the Topological Expressiveness of ReLU Neural Networks
J. Elisenda Grigsby, Kathryn Lindsey
Abstract
Let $F: \mathbb{R}^n \rightarrow \mathbb{R}$ be a feedforward ReLU neural network. It is well-known that for any choice of parameters, $F$ is continuous and piecewise affine-linear. We lay some foundations for a systematic investigation of how the architecture of $F$ impacts the geometry and topology of its possible decision regions, $F^{-1}((-\infty, t))$ and $F^{-1}((t, \infty))$, for binary classification tasks. Following the classical progression for smooth functions in differential topology, we first define the notion of a generic, transversal ReLU neural network and show that almost all ReLU networks are generic and transversal. We then define a partially oriented linear 1-complex in the domain of $F$ and identify properties of this complex that yield an obstruction to the existence of bounded connected components of a decision region. We use this obstruction to prove that a decision region of a generic, transversal ReLU network $F: \mathbb{R}^n \rightarrow \mathbb{R}$ with a single hidden layer of dimension $n+1$ can have no more than one bounded connected component.