Embracing Change: Continual Learning in Deep Neural Networks

Abstract

Modern machine learning excels at training powerful models from fixed datasets and stationary environments, often exceeding human-level ability.Yet, these models fail to emulate the process of human learning, which is efficient, robust, and able to learn incrementally, from sequential experience in a non-stationary world.Insights into this limitation can be gleaned from the nature of neural network optimization, which implies that continual learning techniques could radically improve deep learning as well as open the door to new application areas.Promising approaches for continual learning can be found at the most granular level, with gradient-based methods, as well as at the architectural level, with modular and memory-based approaches. We also consider meta-learning as a potentially important direction. Artificial intelligence research has seen enormous progress over the past few decades, but it predominantly relies on fixed datasets and stationary environments. Continual learning is an increasingly relevant area of study that asks how artificial systems might learn sequentially, as biological systems do, from a continuous stream of correlated data. In the present review, we relate continual learning to the learning dynamics of neural networks, highlighting the potential it has to considerably improve data efficiency. We further consider the many new biologically inspired approaches that have emerged in recent years, focusing on those that utilize regularization, modularity, memory, and meta-learning, and highlight some of the most promising and impactful directions. Artificial intelligence research has seen enormous progress over the past few decades, but it predominantly relies on fixed datasets and stationary environments. Continual learning is an increasingly relevant area of study that asks how artificial systems might learn sequentially, as biological systems do, from a continuous stream of correlated data. In the present review, we relate continual learning to the learning dynamics of neural networks, highlighting the potential it has to considerably improve data efficiency. We further consider the many new biologically inspired approaches that have emerged in recent years, focusing on those that utilize regularization, modularity, memory, and meta-learning, and highlight some of the most promising and impactful directions. A common benchmark for success in artificial intelligence is the ability to emulate human learning. We measure the abilities of humans to recognize images, play games, and drive a car, to name a few, and then develop machine learning models that can match or exceed these given enough training data. This paradigm puts the emphasis on the end result, rather than the learning process, and overlooks a critical characteristic of human learning: that it is robust to changing tasks and sequential experience. It is perhaps unsurprising that humans can learn this way, after all, time is irreversible and the world is non-stationary (see Glossary), so human learning has evolved to thrive in dynamic learning settings. However, this robustness is in stark contrast to the most powerful modern machine learning methods, which perform well only when presented with data that are carefully shuffled, balanced, and homogenized. Not only do these models underperform when presented with changing or incremental data regimes, in some cases they fail completely or suffer from rapid performance degradation on earlier learned tasks, known as catastrophic forgetting. What might be gained by developing neural network models that learn sequentially like humans? First of all, many applications could benefit from continual adaptation to a changing target specification: for example, visual recognition algorithms that need to learn a diverse, growing set of image classes; or household robots that need to incrementally add skills to their repertoire. Continual learning techniques could enable models to acquire specialized solutions without forgetting previous ones, potentially learning over a lifetime, as a human does. In fact, continual learning is generally considered one of the attributes necessary for human-level artificial general intelligence [1.Hassabis D. et al.Neuroscience-inspired artificial intelligence.Neuron. 2017; 95: 245-258Abstract Full Text Full Text PDF PubMed Scopus (569) Google Scholar]. More fundamentally, continual learning methods could offer enormous advantages for deep neural networks even in stationary settings, by improving learning efficiency as well as by enabling knowledge transfer between related tasks. This article will first motivate a taxonomy of continual learning approaches through describing their connections with biological systems. Just as continual learning in humans cannot be reduced to a single biological mechanism, but is rather the product of multiple systems that range from the synaptic plasticity of single neurons to the entire memory system, continual learning in artificial neural networks is not trivial to implement and will likely require a combination of strategies. Before exploring these different approaches in detail, we will delve into the dynamics of gradient-based learning algorithms to understand the challenges of continual learning with artificial neural networks. The study of the natural world and its intelligent species has frequently intersected with artificial intelligence research, including in aspects pertaining to continual learning [1.Hassabis D. et al.Neuroscience-inspired artificial intelligence.Neuron. 2017; 95: 245-258Abstract Full Text Full Text PDF PubMed Scopus (569) Google Scholar]. Biology offers an existence proof for successful continual learning in complex environments and also hints at the design principles and trade-offs of successful approaches. There are multiple mechanisms that enable biological systems to adapt in changing environments without intransigence or forgetting. Thus, in this section we introduce four continual learning paradigms through analogy to their biological equivalents, saving a fuller discussion of each approach for following sections. Additionally, the approaches can be succinctly described by depicting their canonical models, as in Figure 1 (Key Figure). Gradient-based continual learning methods (see ‘Gradient-Based Solutions’) can be understood as distant relatives of models of synaptic plasticity. Mammalian brains have been shown to have complex mechanisms at the synaptic level, which protect against interference between old and new knowledge, or even conflicting facts and skills [2.Wixted J.T. The psychology and neuroscience of forgetting.Annu. Rev. Psychol. 2004; 55: 235-269Crossref PubMed Scopus (639) Google Scholar]. Synaptic plasticity has been studied for decades, with recent studies demonstrating remarkably precise and effective consolidation mechanisms occurring even over very short timescales et a of and and PubMed Scopus Google Scholar]. continual learning approaches have mechanisms on principles et learning with complex et catastrophic forgetting in neural 2017; PubMed Scopus Google but many is paradigm that artificial systems might to enable continual learning (see an it is not that successful in increasingly complex environments is correlated with and of systems. biological brains are with for memory or of to in of and D. et and modular of PubMed Scopus Google Scholar]. systems in the are critical for human learning and are the for memory-based continual learning in artificial neural networks (see for Artificial the complex between synaptic memory, and memory are to be described by it is that memory is the that human learning and adaptation over a of experience et and Scholar]. meta-learning for continual learning (see for Continual is an approach that is by the ability to solutions after experience et to Scholar]. machine learning to the learning approach meta-learning to the of general adaptation that biological systems The of continual learning is by the sequential training and by the from the In contrast to the common machine learning of a or the continual learning on non-stationary or changing environments, often into a set of tasks that need to be This in of or and and as or or it not even have tasks et continual of the on and Scopus Google et for continual et continual learning Scholar]. with learning et of the on Scopus Google et learning for neural of the on the not the the is to the of a are not and a that is to perform household in The cannot be in the and then of the of tasks and the will need to its set over time learning to then and each have a or and require or enable the will need to adapt and also not not is then is learning related tasks and the transfer performance and learning on each and also transfer performance on previous tasks, when of transfer from the the will have to previous tasks as well as to its or In a continual learning to many in Figure and in these that continual learning often between forgetting in a is given an of tasks. This the of which forgetting previous performance can be with a of new experience. and in transfer with the ability to previous tasks. Thus, to an the and how is the to of the and trade-offs can only be we continual learning into a this continual learning in can be and it a to of Continual learning methods catastrophic forgetting is often in research but of the following are critical for many world application to previous tasks. The not have for previous experience it can not with seen in and The approach be it cannot add a new for each catastrophic forgetting and on new tasks not performance on learned tasks adaptation and The be of adaptation to tasks or and of when presented with past plasticity. The be able to learning as new tasks are and a improve related tasks, past and in of learning efficiency and performance learning. The approach not on known or Continual learning methods catastrophic forgetting is often in research but of the following are critical for many world application to previous tasks. The not have for previous experience it can not with seen in and The approach be it cannot add a new for each catastrophic forgetting and on new tasks not performance on learned tasks adaptation and The be of adaptation to tasks or and of when presented with past plasticity. The be able to learning as new tasks are and a improve related tasks, past and in of learning efficiency and performance learning. The approach not on known or networks modern to and consider of data at in fact, this of has to the approach for and applications in the In a learning the is to set the of the network in to some as the between and Gradient-based learning, the most and is an at each a to the in to the a The of this in a each data is to on each to it or we a the that is to each it is or which data or many over many learning to progress and algorithms the of a by a in a direction. The is given by the which can be seen as for each or its will the The are and over a of data a is by a in the of this Figure this The the of the to a as a of the of a this and from a the learning the of the the of and the by a in this direction. many the to a of the we can at learning of a Figure a on the first for and the the of that the learned for has a for this catastrophic sequential training on a set of tasks can to performance on earlier the is so that tasks are by the into a optimization, as shown in Figure we the dynamics that in but the In this the is shown in and the for each are shown in and The from each the its and the is an between the of different tasks. a continual learning the ability to on tasks but it be for a continual learning approach to the dynamics by the or by memory to the with data from seen tasks. Gradient-based algorithms the of a by a in a direction. The is given by the which can be seen as for each or its will the The are and over a of data a is by a in the of this Figure this The the of the to a as a of the of a this and from a the learning the of the the of and the by a in this direction. many the to a of the we can at learning of a Figure a on the first for and the the of that the learned for has a for this catastrophic sequential training on a set of tasks can to performance on earlier tasks. the is so that tasks are by the into a optimization, as shown in Figure we the dynamics that in but the In this the is shown in and the for each are shown in and The from each the its and the is an between the of different tasks. a continual learning the ability to on tasks but it be for a continual learning approach to the dynamics by the or by memory to the with data from seen tasks. a result, for learning to be data be and consider to learn different tasks. the tasks will not on how to set the dynamics will to However, for many one is the will gradient-based learning in tasks are present to the for learning to progress and the The dynamics are the through which gradient-based methods which to the of to learning. Continual learning can be understood as for an to do one that or the dynamics that tasks do not need to be It is how learning in we can tasks are learned at the not the is that most in a are learned and the multiple to learn the et at in deep of the on et training neural networks by in a for deep Scholar]. However, the dynamics require that are even the ones, which has shown that are sequentially, even they are present in the data D. et and their from 2017; and the is for deep models et solutions to the dynamics of learning in deep neural of the on Scholar]. even tasks are complex and presented the might learn sequentially, efficiency to the of continual learning could learning even in stationary learning settings. by the learning dynamics described one promising approach is to the of different tasks. Not only this to the of the it also is by studies of synaptic consolidation in biological brains et a of and and PubMed Scopus Google Scholar]. approach is to the to with from learned tasks D. et memory for continual in et learning with potential methods can be in as for example, in learning, they have the potential to learning in the of conflicting et et for et deep learning models to of the on Scholar]. approaches on the on new tasks to forgetting of previous tasks. methods the of each for previous tasks and to each to this measure et catastrophic forgetting in neural 2017; PubMed Scopus Google et learning through synaptic 2017; Google et learning to of the on Scopus Google et continual 2017; robust of continual et catastrophic forgetting by Scholar]. example, consolidation et catastrophic forgetting in neural 2017; PubMed Scopus Google relies on the to measure the of the with to each and to which most need to be to forgetting. The as a for the of previous tasks, that the by the dynamics is that as et of catastrophic forgetting in gradient-based neural and the training in general et as an for continual et the of training in continual can have of how is set of techniques knowledge et the knowledge in a neural as a to a with to previous tasks. The learning without forgetting approach D. without on and the of previous to be even when learning a new This is by a of the network and previous each new and an to of gradient-based methods is their on of the they to which that they are likely to fail the is presented with many tasks or when the tasks in the are However, these approaches on the they to understand the continual learning and in neural network are a natural and effective to the of interference and catastrophic forgetting in continual learning. offers a between a single which is to and networks for each which catastrophic forgetting but also transfer between tasks (see Figure for of a modular is in biological systems as it of catastrophic forgetting can be by models for different tasks, but this implies transfer between tasks, and and known tasks. is to modular by tasks and with a learning that only a of This is to methods, but with a on which neurons or are for each models often to by new as new tasks are a of network have emerged that add new and et transfer in deep neural and Scopus Google et neural new as learning. solutions add new only when et learning with et deep learning: deep networks to new of the on Scopus Google continual in and then or of the et a for continual of the on et learning neural or learning of new tasks to of a fixed et in neural 2017; catastrophic interference by 2017; et catastrophic forgetting with to the of the on et for continual in et learning with of the on Scholar]. can also as a this can offer and at different of a for a might be modular at a and as well as a and enable learning of sequential tasks. This in earlier in continual learning a first continual Scopus Google (see also of at and has shown with new research et for of the on D. et for learning and transfer in but further study at to understand is from the of which has been from a and efficiency et neural of the on D. et deep neural of the on et neural networks through of the on et training of artificial neural networks with inspired by network PubMed Scopus Google et training very deep of the on training of deep neural networks by dynamic of the on as well as for its to learning and et of for in The on Artificial Google Scholar]. a continual learning et sequential and will in interference and as will be or effective to can also to the of without a modular which we as an important for The paradigm of modularity, when with and offers a powerful to continual learning. into the human-level artificial intelligence will require the ability to and the ability to modular continual learning a Gradient-based and modular methods be for continual learning rather than Gradient-based methods cannot forgetting over and modular approaches can knowledge over they in of neural network the of in different over the of then each after have a that is by as and plasticity in PubMed Scopus Google Scholar]. the sequential experience of by the of a neural network be and A be to the with a from we consider a to the continual learning to implement a neural network memory that can and knowledge or experience. artificial memory is potentially for but it with the of or learning, a for and in a that tasks. The of memory for continual learning is often or the is to a of and then learning from this well as from catastrophic forgetting continual of seen tasks and Scopus Google D. et for continual in In this are also memory methods, which are from methods they not only memory for but also for et learning with et memory in in et of the on Scholar]. and memory are and remarkably effective at but they do not than one can a set of or memory that the of each Not only is this of memory and but it and transfer multiple tasks. the critical in a memory one of which to and this has been the of a of recent et and of the on and Scopus Google et continual learning with in et for continual in et for continual learning et to past knowledge in continual Scholar]. a biological one might that has been in and humans et Full Text Full Text PDF PubMed Scopus Google of or is In memory methods, are models are and then to data as et learning with deep in et of the on Scopus Google et learning with in D. et in Scholar]. we consider methods with learned and the neural et a neural network with dynamic PubMed Scopus Google gradient-based learning to neural networks to from the memory, and to the continual learning a could learn how to and knowledge for and it for and The of the approach a training this of is even in stationary environments or from the neural network suffer from catastrophic forgetting of the memory are for continual learning and the most models have the potential to open new in the but only we can the challenges of training systems. of the solutions mechanisms or for continual learning. a different between as knowledge transfer in a memory-based It is trade-offs can be by learning a from data rather than on human to design a of or approaches have that solutions can be by learning as and learning that need to be et a of and Rev. PubMed Scopus Google to Google Scholar]. meta-learning approaches can be described as timescales of an that on tasks and an that performance over multiple to continual learning, can be that for performance in non-stationary et with of the on et learning continual adaptation for of the on et to Scholar]. example, meta-learning for continual in that learn on correlated of this the and could in catastrophic the the to forgetting and improve the non-stationary the to an of of the of meta-learning, is they are and for effective continual algorithms can be understood in of adaptation at different time of learning, or the necessary for an of The of the is to improve adaptation in different data are training and to at time algorithms can be by their or of the which adaptation to tasks, or the which a of In et to learn by by in et meta-learning for adaptation of deep of the on can also be et networks for one in et networks for in neural et to et to learn Artificial Scopus Google as a for of the on et in et neural of the on or approaches et adaptation of the on et with of the on et learning of the on et with of the on and Scopus Google Scholar]. In these the to improve the of adaptation to a new algorithms can be understood in of adaptation at different time of learning, or the necessary for an of The of the is to improve adaptation in different data are training and to at time algorithms can be by their or of the which adaptation to tasks, or the which a of In et to learn by by in et meta-learning for adaptation of deep of the on can also be et networks for one in et networks for in neural et to et to learn Artificial Scopus Google as a for of the on et in et neural of the on or approaches et adaptation of the on et with of the on et learning of the on et with of the on and Scopus Google Scholar]. In these the to improve the of adaptation to a new sequential can be considered as a meta-learning et of the on et of the on et in et adaptation and knowledge a new approach to continual in Scholar]. in et continual learning this can be for continual learning. In the is an that a that can tasks However, that in natural are many of tasks that are or on the on the or the of the the meta-learning the emphasis to adaptation and rather than this is not and on the tasks There are that one cannot to for of or or in is for of the of the experience a is an of an experience that not need the of meta-learning new solutions for continual learning is this has to be as well as design of the et a of datasets for learning to learn from few et a benchmark and for and Scholar]. meta-learning can some of the in machine learning, this is by the tasks past these meta-learning could continual learning research the of knowledge with a of learning in non-stationary learning often to the ability of humans to learn and a from a few However, we do not often on the ability of humans to learn over a of and it is this that human of and This article not only to highlight the of continual learning, but also to the of modern neural networks in this in the that in an gradient-based the we have learning paradigms that have the potential to be impactful to Not these paradigms have in neuroscience and biological systems. Gradient-based approaches the of neural networks and have been shown to catastrophic forgetting. offer solutions to interference and catastrophic enabling transfer through of skills and memory models could be a for learning, and meta-learning approaches could algorithms and potential for it is important to also the in machine learning models that of and cannot be in Continual learning could these through improving the of the learning and through developing that that or are we an to that the ability of gradient-based approaches to learn complex tasks well the dynamic and improving the efficiency of are the trade-offs between the different and can we the trade-offs of biological these are are the different or of continual learning that we do we the need to the ability to facts or is the modular of a continual learning has to be than it might it over can we learn how to or to continual might we that these mechanisms are robust to in data when with models that learn how do we that are do we and we learn the to continual we do this at the data and learning to learn the as as the the learned be robust or we an to that the ability of gradient-based approaches to learn complex tasks well the dynamic and improving the efficiency of What are the trade-offs between the different and can we the trade-offs of biological these are are the different or of continual learning that we do we the need to the ability to facts or is the modular of a continual learning has to be than it might it over can we learn how to or to continual might we that these mechanisms are robust to in data when with models that learn how do we that are do we and we learn the to continual we do this at the data and learning to learn the as as the the learned be robust or the ability to transfer knowledge from a to improve performance on a learned a in neural networks learning a new performance on previous tasks. how different in a neural network are for the of the the of the of a of tasks that learning is and efficiency on the following tasks. given by the the of the are by a in this the will the ability to transfer knowledge from previous tasks to improve performance and learning efficiency on a related the of the with to the of the the that learning to in to improve given a continuous a the in the is the of the the the and the as this only the is the nature of the as one can only a in the by how the an of the of the in modern machine learning. the set of by a machine learning in to as the or the target a that a to learning, as the between and a of or with and which process some given and an It is on neurons in biological describing a process or with a has many This can be the for network or for the implies that few are each learning

Topics & Concepts