Can We Read Neural Networks? Epistemic Implications of Two Historical Computer Science Papers
Fabian Offert
Abstract
This review looks at two technical papers from the field of computer science that, at the time of writing, should be considered historical. Although their respective technical approaches have since been replaced with newer, better, and more efficient ones, when looking back through the lens of critical AI studies they mark the beginning of a type of theoretical reflection within computer science that distinctly links technical machine learning research to research in the humanities.Machine learning models are cultural artifacts. They are trained on (limited) real-world data and often designed to make decisions with real-world impacts. The relation of a machine learning model to the world is thus a relation of interest. What kind of representations do machine learning models produce? As the world is necessarily mirrored in a machine learning system to some degree, as there exists, with Walter Benjamin, an approximation of a mimetic faculty, what are the modes of representation that a machine learning system has at its disposal? Is it possible, in other words, to “read” a machine learning model? Can we, as humans, rely on our capability to decode systems of representation, such as artistic descriptions of the world in text or image, to understand neural networks?The two technical papers at the center of this review shed some light on this fundamentally humanist question. Concretely, they suggest that we are currently witnessing a turn toward postsymbolic computation, a paradigm under which nothing is language and everything is language at the same time. In that sense, both papers could serve as a starting point for a renewed methodological discussion within critical AI studies and related fields of study on the legibility and interpretability of machine learning systems.“Intriguing Properties of Neural Networks,” written by Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus, describes two “counter-intuitive” (1) properties of deep convolutional neural networks (CNNs): their susceptibility to input perturbations (termed adversarial examples),1 and the entanglement of learned concepts in their internal structure. It is the latter “intriguing” property that is of interest here.CNNs usually consist of tens of thousands of atomic units called neurons, arranged in a layered fashion.2 Before training, these neurons pass on information arbitrarily. If we imagine a binary (i.e., two-category) image classification network, this would mean that an input image is randomly assigned to one of two classes. Training the neural network, then, means adjusting all neurons such that, eventually, the networks can make correct classification decisions. This is usually achieved by “showing” the network lots of example images and automatically adjusting the neurons after each iteration. Once a network is fully trained, all neurons play a specific role in the network’s decision process. Even if a neuron’s role is just to stop the information flow (i.e., to pass on zero values to the next layer), these one-way streets are in no way less relevant to the accuracy of the whole system than are any other neurons.3What Szegedy et al. show experimentally is that one of the implications of this entanglement of neurons is an entanglement of concepts: rather than developing hierarchical modes of representation, CNNs tend to develop idiosyncratic modes of representation (see Geirhos et al. 2020; Offert and Bell 2021) where it is hard if not impossible to define which parts of a network represent which parts of a real-world object.4 For instance, if we consider a CNN that has learned to distinguish between two real-world objects, say, dogs and cats, the part of the network that is important in recognizing dog ears might equally be important in recognizing cat tails. In a way, established concepts (dog, cat) are thus dissolved when they are learned by a CNN, as there is no way to reconnect them to the real-world objects they represent. It is important to note that this makes the neural network not less but more successful at its task of distinguishing between dogs and cats; the success is simply not tied to any human way of solving it.Hence, we cannot trust neural networks to represent the world in a way that is coherent to us. Szegedy et al. for the first time in the technical literature show that machine learning models are fundamentally alien, not in an exaggerated phenomenological sense but in the sense that they require empirical study. In their (with Gilbert Simondon) fully concretized, final form, neural networks start to resemble natural rather than technical objects. Like compiled computer programs that need to be painstakingly “disassembled,” neural networks require extensive arrays of “interpretability” techniques to become legible. Unlike those programs, however, there is no code to fall back to, no symbolic description of the same process that could be referenced. Trained machine learning models, in other words, usher in a new paradigm of postsymbolic computation.“Learning to Execute,” written by Wojciech Zaremba and Ilya Sutskever, is another marker of this paradigm change. It demonstrates that language models, specifically sequence-to-sequence models based on recurrent neural networks, can, within certain limits, “learn to execute” short computer programs; that is, they can infer from a short program provided as input what the program would output were it to be run. The conditional is important here: the program is, of course, not run (this would defeat the purpose of the experiment), but the problem of running a program is treated as the problem of translating textual input (a program) into textual output (the predicted output of the program were it to be run). Zaremba and Sutskever show that this is indeed possible, at least within the framework of their limited experimental setup. What they implicitly suggest is that neural networks could approximate general-purpose computation if they only learned to “read” properly. They could become fully functioning replacements for the computers they are themselves running on by treating literally any computational problem as a problem of translation.The implications of this understanding are far-reaching and lead directly into the current debate on “foundation” models. These models, mainly thought as very large transformer-based large-scale language models (LLMs) like GPT-3 (Brown et al. 2020) or PaLM (Chowdhery et al. 2022), are meant to all but replace hand-coded general-purpose computing with “prompt”-based, natural language description. Already today they can indeed execute (i.e., infer the meaning of) programs, solve mathematical equations, and demonstrate commonsense reasoning (Chowdhery et al. 2022).We have thus arrived at the inverse of the situation produced by image models like the one described in Szegedy et al.: for LLMs, everything is text, everything is forced into an already existing (human) mode of representation. Concretely, formal languages (e.g., programming languages) are treated like natural languages. While it looks like this approach “just works,” the epistemic implications are entirely unclear. Can we even imagine a mode of representation for mathematical operations that is not precise yet still produces precise results? Extrapolating the comparatively minor dangers of current-generation models (which are still in an advanced prototype stage), what would it mean for all computation to become natural language processing?These are speculative questions, of course, but they point to a looming paradigm change with great significance for the humanities. I suggest that we can understand this emergence of idiosyncratic modes of representation in neural networks, on the one hand, and this making readable of everything by means of neural networks, on the other, as two sides of the same paradigm change, from symbolic to postsymbolic computation. The two papers discussed here are early symptoms of this paradigm change, which is only accelerating. One of the tasks of critical AI studies and related fields of study, then, is to develop a critical methodology that can accommodate a new reality of postsymbolic computation. Importantly, this implies a move away from the universal narrative of “the binary” as the root for all evil, and an acknowledgment that injustices exist even where systems pretend to be “readable,” where the explicitly machinic is replaced with familiar human modes of representation.